Complete Guide to

App Store & In-App A/B

Testing:
Proven strategies, professional tips, and
best practices

This book is the result of cooperation between ​SplitMetrics​ and ​Apptimize​.

Our A/B testing platforms ensure success of mobile apps and best possible digital
experience.

www.splitmetrics.com
www.apptimize.com

Author and Project Editor: Liza Knotko

Author: Thu Phan
Copy editing: Anton Tatarinovich

Thanks to Sylvain Gauchet, Gabe Kwakyi, Valeria Shytikova, Natalie Ostapovich and
other experts for sharing their A/B testing insights.

Copyright © 2018
SplitMetrics Inc. All rights reserved.

Please don’t share this book without giving us appropriate credit and a link.
For further information, contact [email protected]

2
Table of contents

Introduction

Part One: App Store A/B Testing

Chapter 1. Deconstructing App Store A/B Testing

Chapter 2. Workflow: A/B Testing Timeline

Chapter 3. Pages to Optimize in App Stores

Chapter 4. Most Influential Page Elements for A/B Testing

Part Two: In-App A/B Testing

Chapter 5. In-App A/B Testing Importance for Long-term Product Growth

Chapter 6. In-App Testing and Optimization Best Practices

Chapter 7. Building vs. Buying Your A/B Testing Solution

Chapter 8. Mobile Retention 101

Final Thoughts

3
Introduction

A/B testing has been around in the marketing land for quite a while. One of the
best tools for getting data on target users’ behaviour - it’s been used across
multiple categories, from websites and mobile apps to user acquisition on major
app stores.

Now, let’s imagine something.

What if you could know which app features your users will love before starting
to develop them?

What if you could get into your users’ heads and learn which types of texts and
pictures appeal to them most, use this information on your app pages and inside
the app itself, and boost your marketing ROI?

What if ASO was no longer a guesswork, but a series of data-backed actions

towards proven results?

All of that is what A/B testing is used for by thousands of app publishers around
the globe. There’s no rocket science - A/B testing can’t be called a novelty and
might’ve been already used by your marketing team.

That is why we created this guide - to give you actionable insights and proven
tactics, which will help you increase conversion on app stores and keep users
engaged inside the mobile app via A/B testing. You’ll learn to identify the right
elements to test, get the complete roadmap of necessary actions, get tons of
benchmarks and much more.

Let’s jump in.

4
CHAPTER 1

Deconstructing
App Store A/B
Testing
App Store A/B Testing Overview
How to Run Valid A/B Tests

- Research Prior to A/B Testing

- Hypothesis
- Traffic
- Targeting
- Finishing A/B Tests and Interpreting Results
- Measuring Impact of A/B Testing Results in Stores

​Pre-launch A/B Experiments

5
​ Chapter 1. Deconstructing App Store A/B Testing

I. App Store A/B Testing Overview

A/B testing (often referred to as split testing or bucket testing) is a means of comparing
two options and analyzing results. It can be used to test, compare, and analyze just
about anything.

It is most often associated with websites and recently, mobile apps; A/B testing is
primarily used for conversion rate optimization.

Online businesses regularly optimize their landing pages and improve ROI through the
help of A/B experiments.

If you want to increase your app page conversions on the App Store or Google Play, it’s
vital to understand that a solid ASO strategy can help you get impressive results. Smart
A/B testing should be an indispensable ingredient of such strategy.

6
​ Chapter 1. Deconstructing App Store A/B Testing

A/B testing of various product page elements is a really efficient way of optimizing your
app’s conversion rate.

But remember that running your tests mindlessly will give no results, that’s why you
should take into consideration quite a few aspects which predetermine success or
failure of your experiments.

II. How to Run Valid A/B Tests

The truth is if you are not ready to spare time for a thorough preparation, the chances
of your experiment’s success shrink to zero. So it definitely makes sense to set aside
some time for:
- research;
- hypothesis elaboration.

Let’s dwell on each of 6 key aspects of successful and high-performance A/B testing.

1. Research Prior to A/B Testing

The better you do your homework, the more chances your test will give meaningful
results. So take enough time to do the research and analyze the market, competitors,
target audience, etc.

You can start with the following steps:

1. study app pages of your competitors carefully;
2. determine trends and best practices in your app category;
3. take a closer look at the industry leaders;
4. think how you can reflect the best practices in the design of your product page
elements.

Another aspect you should think over properly is your target audience. Having a clear
portrait of your ideal users is a must. You should know everything about them from
basic things like age and sex to their interests and preferences.
These insights will help to adapt your future app store creatives and ad campaigns.

7
​ Chapter 1. Deconstructing App Store A/B Testing

PRO TIP
Constant A/B testing helps to perfect your target audience portrait if you
create ad groups with different audience settings to check which one
performs better.

The research stage also includes the qualification of traffic sources. It’s really important
to discover ad channels that bring more loyal users. You can use A/B testing for such
qualification as well launching campaigns with the identical targeting in different ad
networks and comparing their performance later.

Determining the most effective ad channel doesn’t only makes your

A/B experiments more meaningful, this knowledge is essential for any marketing
campaign in general.

According to the experience of SplitMetrics clients, Facebook and Adwords proved

themselves as the best traffic sources while other ad channels have major
shortcomings such as:

● loads of filtered visitors due to overall low quality of traffic and fraud;
● in case of cross-promo, you need way more traffic to achieve statistically
significant results;
● using Smart banners for A/B testing, you’ll have to run your experiment for a
very long time (average test duration is up to 1-2 months).

2. Hypothesis for A/B testing

Only at fulfilling the above-mentioned steps, you can proceed to hypothesis
elaboration. It is impossible to overstress the importance of a solid hypothesis for A/B
testing.

Keep in mind that thoughtless split-experiments are doomed from the very beginning.
A/B tests for the sake of A/B tests make absolutely no sense. So it definitely makes
sense to take time and mull over a solid hypothesis worth proving.

8
​ Chapter 1. Deconstructing App Store A/B Testing

But what makes a good hypothesis? Remember that hypotheses are bold statements,
they can’t be formulated as open-ended questions. A neat hypothesis should say what
you plan to learn from an A/B experiment.

Any A/B testing hypothesis should contain the following elements:

1. a proposed solution (what do you plan to change);

2. the anticipated results of implementing the solution;
3. hypothesis rationalizing (this element is highly recommended but not
obligatory).

Here is an example of a good hypothesis based on the abovementioned model

formulated by Rovio prior to Angry Birds 2 launch:

If I change the orientation of the screenshots from horizontal to vertical, my app

will get more installs as my target users are not hardcore gamers and are not used
to landscape mode.

9
​ Chapter 1. Deconstructing App Store A/B Testing

By the way, the experiment based on the above-mentioned hypothesis yielded the
impressive 99.7% confidence level and demonstrated curious results. Indeed, portrait
app screenshots did better in terms of conversion which is, to some extent, against the
industry standard​s. The hypothesis was proven right.

Keep in mind that it’s useless to A/B test practically identical variations. Don’t expect
stellar results running an experiment for two icons for your game depicting the same
character with different head posture.

What can be tested in this case then? Here are a few ideas:
- Various characters.
- Background color.
- Character’s facial expression, etc.

Let best practices you found out during the research be your inspiration for the test
you can run. A solid hypothesis will become a basis for further goals setting and
choosing the experiment type.

For instance, you decided to test screenshots orientation. Now you have to decide
what kind of screenshots experiment you’d like to launch:

● On your product page​​: the results of such test will help to boost overall
conversion of your product page. It’s extremely important if you mainly rely on
high volumes of paid traffic.

● In the context of the search results: this experiment type will show how your
screenshots perform next to competitors. It’s essential for apps which stick to
organic traffic and strive for better discoverability.

Now you are ready to design variations for testing and launch an A/B experiment.
Thanks to such specialized A/B testing platforms as SplitMetrics, starting a test is a
matter of minutes.

10
​ Chapter 1. Deconstructing App Store A/B Testing

3. Traffic for A/B Testing

It is essential to understand that no A/B experiment succeeded without decent and
well-considered traffic. The truth is that you may have awesome creatives and use the
best A/B testing platform, but the poor quality and insufficient volume of traffic will
predetermine the failure of your A/B experiment.

Running an A/B test, you should reach a really high ​confidence level before considering
the experiment finished. Confidence is the statistical measurement used to evaluate
the reliability of an estimate. For example, 97% confidence level indicates that the
results of the test will hold true 97 times out of 100

How much traffic do you need to run a meaningful experiment? Lots of publishers ask
this question. Unfortunately, nobody will tell you the gold standard number which will
guarantee high confidence level.

The necessary volume of traffic (​sample size​) is highly individual and depends on
myriads of factors such as:

● Traffic source. ​Different traffic channels have different conversion

potency. You’ll need less traffic from, let’s say, Facebook than from
some strange ad network swarming with bots.

11
​ Chapter 1. Deconstructing App Store A/B Testing

● The conversion rate ​of the app’s product page. The better the
conversion, the faster you reach the required confidence level.

● Targeting. This comes from the previous factors – trustworthy traffic

sources provide great targeting options which fuel good conversion of
the product page under test.

According to SplitMetrics observations, it normally takes at least 50 ‘Clicks to Install’

per variation to get statistically significant results. On average, you need 400-500
visitors per variation for a test on your app’s product page and 800-1000 users per
variation if you run an experiment on the search or category page. These numbers can
help you take your bearings, but again, this index is highly individual.

PRO TIP
It’s really important to remember that the smaller the difference between
the conversion rate values of the variations A and B, the greater sample size
is required for the test. So it’s better to use one of the specialized
calculators to estimate a sample size of traffic for A/B testing. For example,
one by ​Evan Miller​.

4. Targeting for A/B Testing

As it was mentioned before, all factors are interconnected and require your close
attention. However, one of the most impactful things you should consider is targeting.
Smart targeting helps to get statistically significant results buying less traffic.

That’s why it’s so important to study your primary audience in the course of the
research phase. The more granular your targeting, the better results you get. Lots
publishers prefer to drive traffic from Facebook due to its amazing targeting options.

To ensure trustworthy and quality test results of A/B testing, it’s important to filters
out:

12
​ Chapter 1. Deconstructing App Store A/B Testing

● users that don’t meet targeting requirements (e.g. Android users visiting iOS
experiments);
● bots and fraudulent traffic;
● repetitive visits;
● mistaps and misclicks, etc.

That’s why it makes sense to rely on specialized A/B testing tools like SplitMetrics
which won’t let you fill your experiments with absolutely irrelevant traffic thanks to
built-in filters.

5. Finishing A/B Tests and Interpreting Results

We already made it clear that confidence level is one of the most important metrics
which helps to determine when to finish your experiment. Yet, it’s not the only one.

Here is the list of metrics to consider before finishing your A/B test:

● Visitors per variation. The total number of users that visited the variation
of the product page (on average at least 400-500 users per variation for a
product page test and 800-1000 visitors for a search/category test)

● Clicks to install per variation. The percentage of visitors that tapped the
Install button on the variation page (minimum 50 installs per variation).

● Time to click to install. How long visitors stayed on the variation page
before tapping the Install button.

13
​ Chapter 1. Deconstructing App Store A/B Testing

● Engagement rate. The percentage of visitors that interacted with the

variation page. Interactions can include page scrolls, taps on Install button,
and ‘Read more’.

PRO TIP
Ensure that your experiment lasts for 5-10 days before finishing it. This way,
you’ll capture users’ behavior during all weekdays making the results even
more accurate.

Once you finish your experiment, the fun part begins. You find out whether your
hypothesis was wrong or right and try to find reasons behind the obtained results.

Above all, it’s incredibly important to understand that there’s no such thing as negative
results of A/B testing. Even if your new design suffers a crushing defeat from the old
one, the experiment prevents mindless upload of these new creatives to the store.

Thus, an A/B test with negative results averts mistakes that could cost your app
thousands of lost installs.

There’s one more metric we should consider before deciding to use a winning variation
in the App Store.This metric is called ‘Improvement’, it shows increase or decrease in
the conversion rate of the variations in comparison to the control one.

14
​ Chapter 1. Deconstructing App Store A/B Testing

If an A/B experiment shows that your new design wins by 1% or less, don’t rush to
upload it to the store straightway. Remember that only the advantage of at least 3-4%
can really impact the conversion of your app’s store product page.

Word to the partner

Zimad​ (an international developer and publisher of games for all major app stores):

“We noticed that if our new app store screenshots set wins a test with a little
advantage (around 1% difference in the conversion rate), that wouldn’t guarantee
the gain of conversion in the real environment. It may even result in the conversion
rate decrease sometimes.

Before noticing it, we were surprised that we hadn’t seen any positive changes. So
make sure that your winning set of app store screenshots overcomes the control
one by at least 2% to get the real visible benefits.”

There are cases when publishers draw quality traffic to their experiments but the
confidence level refuses to grow. It normally happens because of a similar performance
of the variations due to weak hypotheses.

PRO TIP
Keep in mind that even if your variations look totally different, it doesn’t
prove the strength of your hypothesis. In such a situation, it’s highly
recommended to finish the experiment and revise creatives.

Prepare yourself to the fact that your assumptions will be wrong as a rule. Yet, finding a
game-changing product page layout definitely worth the effort.

15
​ Chapter 1. Deconstructing App Store A/B Testing

6. Measuring Impact of A/B Testing Results in Stores

Sustainable results that change your conversion rate for the better is the ultimate goal
of experimenting with your app’s product page unless you are an A/B testing enthusiast
and run such tests for sheer delight.

Many publishers wonder how to measure the impact of test results in the main stores.
It is indeed an interesting yet tricky question. For sure, you should have a finger on the
pulse of your conversion fluctuation after the upload of a new product page version.

For instance, in the graphic below, you can see how the conversion rate of Zimad’s
Magic Jigsaw Puzzles changed after they optimized their app product page for Japan
with help of A/B testing. Optimization of screenshots with SplitMetrics resulted in ​36%
conversion rate increase​ which was obvious from their iTunes Analytics.

Nevertheless, remember that any app store is a compound ever-changing environment

teeming with factors influencing your app’s performance. To see a positive impact of
your A/B testing results you should always be consistent.

For instance, let’s imagine you tested your new screenshots on users from Hong Kong.
You got stunning results and uploaded them to the US store, as it’s your core market.
You expect the same conversion boost you saw during the test but nothing really
happens. It’s easy to claim A/B testing worthlessness in this situation, but targeting
inaccuracy is to blame here.

16
​ Chapter 1. Deconstructing App Store A/B Testing

The same is true when you apply the results of testing in the Google Play to the App
Store failing to acknowledge the difference between Android and iOS users, their
unlike perception of the store, varying behavioral patterns, etc.

The main recommendation here is to be diligent and consistent. Thus, you’ll see app’s
performance improvement in iTunes or Google Console without extra effort.

PRO TIP
Test only 1 element and only 1 change at a time as experimenting with
different concepts at once, you will probably get ambiguous results requiring
multiple follow-up tests.

Providing you are ready to take time and use your wits, split-testing testing will become
a safeguarded tool in your ASO arsenal. A/B experiments don’t only boost apps’
performance on both paid and organic traffic, they also provide essential metrics and
analytics which can and should be applied in your audacious marketing activity.

III. Pre-launch A/B Testing

Pre-launch phase is one of the most critical periods in the lifecycle of an app. Publishers
have to deal with myriads of tasks which put their time management and priority
setting skills to the test.

However, an overwhelming majority of app marketers fail to realize that skillful A/B
testing can smooth things down for them and assist in various pre-launch tasks: from
product ideas validation to ensuring stellar conversion rate when an app is not even
live.

In fact, the split-testing functions we are going to discuss below can redefine your
pre-launch strategy saving time and contributing its efficiency.

17
​ Chapter 1. Deconstructing App Store A/B Testing

Validating Product Ideas

A/B testing is at hand when it comes to qualifying ideas for an app. Thus, developers
can track users reaction to various concepts, characters, features,messages, marketing
banners, etc. and maximize their CVR.

This evaluation of ideas saves time and budget giving publishers a unique chance to
give up doomed undertakings or change the direction of their app marketing and
development.

Developing the Right Positioning

Apps usually have quite a few features, so it’s hard to understand which one will
resonate most with your target audience. A/B testing helps to leave all doubts behind
and experiment with various messages highlighting different features.

The photo and video filter app MSQRD ran a ​series of A/B tests to identify the most
appealing mask from the range this app planned to offer.

The monkey mask won as 2016 New Year’s Eve was approaching and it’s the year of
the monkey according to the Chinese zodiac. The team placed this filter first in their
screenshot set ensuring it’ll be the first thing potential customers would see.

18
​ Chapter 1. Deconstructing App Store A/B Testing

The right positioning from app’s early days means not only loyal customers, but also an
impressive performance on both paid and organic channels.

Product Pages Reinforcement

All store product page elements (from name to screenshots) have their role in forming
users’ perception of an app. These app page pieces have an enormous impact on
conversion. Nevertheless, lots of marketers neglect their importance especially when
an app is not in a store yet.

Even analytical minds that wouldn’t alter a single keyword without a proper research
tend to forget that app’s product page is an ultimate decision-making destination. Icon,
screenshots, description, etc. have to represent the essence of your app just like
keywords or even better.

Trusting instincts is not enough. Unfortunately, it’s standard practice to put diligence
aside and rely on a subjective opinion of your team members ignoring available app
store optimization options. A/B testing makes you leave all guessing games behind and
be guided by statistically significant numbers.

For instance, Rovio took advantage of this gimmick and ​played around screenshots
before Angry Birds 2 release (we’ve already mentioned one of their hypothesis which
busted industry standards). A series of app screenshots and other experiments
produced the assets combination that generated a maximum conversion exceeding
others by 13%.

19
​ Chapter 1. Deconstructing App Store A/B Testing

It is a sound improvement that meant millions of extra downloads for Angry Birds 2.
The game got more than 20 million downloads during the first week after its release
and A/B testing with SplitMetrics can be credited for at least 2.5 million downloads. For
a hit game like that, it is a huge cost cut.

20
​ Chapter 1. Deconstructing App Store A/B Testing

Identifying Ideal Audiences and the Best Ad Channels

Having a clear vision of your ideal target audience is a core element of any app
successes. The sooner you identify who your customers are, the better. A/B tests assist
in solving this problem even when your app is not in the store.

You can find out who is more likely to install your app running experiments on different
demographic groups. These data are vital for any further marketing activities once your
app is live. For example, G5 Entertainment ran a series of experiments for their ‘Hidden
City’ app and found out that their most converting targeting is a 35+ female who loves
board games and is interested in puzzles, riddle, and mysteries.

21
​ Chapter 1. Deconstructing App Store A/B Testing

PRO TIP
Professional A/B testing platforms like SplitMetrics allow to collect potential
users contacts within such pre-release A/B testing. Thus, you can building
early adopters list for company’s newsletter and app updates.

A/B testing is indispensable for ad channels qualification as well. Discovery of an ad

source which brings a lot of loyal users advances any marketing game plan. App
publishing companies check the performance of different ad channels via split-testing.
At comparing their performance, they opted for one of the variants for the promotion
of their new games and apps.

User Behaviour Analysis

Using scroll and hit maps of ​app A/B testing platforms, publishers get precious insights
on users interaction with app’s product page clearing up what can be altered for the
sake of better app performance.

Moreover, such pre-launch A/B testing analytics assists in uncovering app page
bottlenecks. Developers have a chance to determine potential problems of their app
and points of growth in a short space of time.

For instance, Darby (an app with

DIY videos) discovered that their
icon misled customers. A “play” sign
in the icon made audience believe
Darby was a video editing app. A/B
tests helped to nip this problem in
the bud and the icon was altered
completely.

22
​ Chapter 1. Deconstructing App Store A/B Testing

Pricing policy
This point is extremely important for aspiring paid apps. It’s vital to identify the price
which won’t scare potential users and will help to earn more eventually. A/B testing
different prices before the app’s launch is one of the best options. It may even result in
change of app’s policy and waiver of paid form in favour of free model with in-app
purchases.

Localization for Other Markets

Pre-launch phase or period before major app redesign is really favourable for localizing
your app. However, it’s not enough to merely translate description, and you should
localize beyond the text. It’s important to understand that you’re adapting your
product for another culture, not just another language. A/B experiments are handy to
test various cultural hypotheses.

For instance, FiftyThree used split-testing to localize their Paper app for the
Chinese-speaking market. Renewed screenshots in Chinese with multicolored
backgrounds had 33% better conversion than the English ones.

23
​ Chapter 1. Deconstructing App Store A/B Testing

There’s no need to give yourself a hard time trying to guess what will work better for
your app prior its launch. Using A/B testing, you can empower your conversion even
when app isn’t even live. Thus, you’ll ensure stellar results right from the start of your
app’s life in a store.

Split-testing doesn’t only takes conversion rate to the brand new level, it also facilitates
decision-making process making it transparent and eliminating unnecessary team
conflicts. Plus, using specialized A/B testing platforms, marketers also get valuable
insights on users behaviour which can be used for further development of an app and
store page polishing.

24
​ Chapter 1. Deconstructing App Store A/B Testing

KEY TAKEAWAYS

1. A/B testing of various product page elements is a really efficient way of

optimizing your app’s conversion rate.

2. Running your tests mindlessly will give no results, that’s why you should
take into consideration quite a few aspects.

3. The key factors which predetermine A/B testing success are:

a. Research;
b. Hypothesis elaboration;
c. Traffic;
d. Targeting;
e. Timing;
f. Post-experimental analysis.

4. Pre-launch A/B testing can empower such tasks as product ideas

validation, right positioning development, user behaviour analysis,
product pages reinforcement, localization, identification of ideal
audiences and the best ad channels .

5. It’s better to opt for specialized A/B testing platforms like SplitMetrics to
ensure quality results and minimize inaccuracy.

25
CHAPTER 2

Workflow: A/B
Testing Timeline
1. Research and Analysis;
2. Brainstorming Variations;
3. Designing Variations;
4. Running Your A/B Experiment;
5. Evaluation of A/B Testing Results;
6. Results Implementation and Tracking;
7. Follow-up Experiments.

26
Chapter 2. Workflow: A/B Testing Timeline

Workflow: A/B Testing Timeline

When you’re an A/B testing newbie, it might be hard to know where to begin. A/B
testing is not as straightforward and easy as it may seem at first. Especially, when
you’re aware of all aspects which should be considered that we’ve mentioned above.
However, there are certain steps which are integral parts of any A/B testing activity.

Let’s review each step of a classic A/B testing workflow. Mind that pretty much any A/B
testing timeline should include the following action points:

1. research and analysis;

2. variations brainstorming;
3. designing of variations;
4. running your A/B experiment;
5. results evaluation;
6. results implementation and tracking;
7. follow-up experiments.

1. Research and Analysis for A/B Testing

Duration:​​ 2 days – week

Let’s make it clear – leading companies don’t A/B test random changes these days.
They’d rather use research and elaborate strong hypotheses based on the collected
data. And, as we know, a solid hypothesis is a cornerstone of any A/B testing success.

Word to the partner

Here’s the list of core research activities made by Gabe Kwakyi from ​Incipia that can
help you embark upon this highly important phase.

a) ​Compare your screenshots, feature graphic, videos, and icon designs with the styles
of other apps that appear near yours in keyword search results. Your app has to stand
out among these apps in order to earn a user’s initial install.

27
Chapter 2. Workflow: A/B Testing Timeline

It’s preferable to focus on the keywords that have the highest volume and for which
your app ranks best for, as these keywords will have the highest contribution margin
to your organic downloads.

b) ​Compare the messaging found in your

captions, videos, and descriptions against
that of your competitors. You are to come
up with the messaging that gives your app
a competitive advantage.

Think over what statements you can make

that your competitors cannot. Analyze the
advantages your rivals possess and how
you can overcome them in order to
convince users to download your app.

с) ​Read your app’s reviews to determine

the language and key features that users
care about and integrate it into your
messaging, mark selling points that don’t
appeal to your users and are basically the
dead weight of your product page.

Try to focus on features that are marked

most useful by users, inspect the most
recent reviews for better results.

d) ​Review your app’s usage data to figure

out which features or configurations are
most-used or most-purchased. It makes
sense to use these insights to increase your
app listing’s appeal to new users.

28
Chapter 2. Workflow: A/B Testing Timeline

2. Brainstorming Variations for A/B Testing

Duration: ​a couple of hours

Your variations for A/B Testing should always reflect a quality hypothesis. Form your
hypothesis and create variants after researching how your app can be best positioned
next to your competition.

Keep in mind that hypothesis is not a question, it’s a statement which suggests a
solution, its expectation effect, and its rationalization. For example, ​Prisma decided to
test the following presumption based on the app category best practices:

Using a bolder font in captions placed on the top of screenshots triggers

conversion growth due to better readability.

29
Chapter 2. Workflow: A/B Testing Timeline

Thus, on this step, you’d also want to think over variations layout based on the
hypothesis you plan to test and prepare technical design specification.

3. Designing Variations for A/B Testing

Duration:​​ 1 day – a few weeks

It’s the phase where designers create layouts for the upcoming A/B testing. The design
is to reflect presumptions under the test and correspond to the technical design
specification.

The duration time of this phase depends on the workload of your in-house designers.
Companies that don’t have a staff designer have to resort to third-party solutions
which, naturally, takes more time.

4. Running Your A/B Experiment

Duration: ​7-14 days

You can proceed with A/B testing itself once the first 3 steps are completed. It’s vital to
choose a traffic source and appoint a target audience before launching an experiment.
Remember that users are to be split equally.

A/B test won’t even bother you if you decide to test with specialized tools like
SplitMetrics as they distribute audience members automatically landing these users on
two different variations. All you have to do is fill your experiment with quality visitors
and the platform will do everything else for you:

● collect powerful insights into visitor behavior on your app page;

● filter fraudulent traffic to ensure results trustworthiness;
● calculate confidence level sufficient for determining the winner of your
experiment.

30
Chapter 2. Workflow: A/B Testing Timeline

Relying on quality A/B testing platform, you eliminate human errors and ensure the
highest possible quality of your experiments.

It’s recommended to run split-tests for at least 7 days to keep track of users behavior
on all weekdays. This contributes results trustworthiness as well.

5. Evaluation of A/B Testing Results

Duration:​​ a few hours

You can finish your test and start analyzing its results at reaching a statistically
significant number of active users who visited the experiment. Upon reaching the
trustworthy confidence level, your original hypothesis will be either proven or refuted.

31
Chapter 2. Workflow: A/B Testing Timeline

It’s worth repeating that a disproved hypothesis doesn’t mean that your A/B testing
ended in a fiasco. Quite the opposite. Negative results prevent us from making changes
that could cost us thousands of unseized installs.

PRO TIP
Conversion changes analysis is a must, there’s no denying. Yet, don’t forget to
spare time to explore other metrics which will help you understand your users
better and empower your further marketing activity with these insights:
● video metrics;
● scroll heatmaps;
● time on page;
● interaction funnel;
● screenshot stats;
● behavioral segments.

32
Chapter 2. Workflow: A/B Testing Timeline

6. Results Implementation and Tracking

Duration:​​ 1-2 months

Providing you find a clear winner, you can implement the results in the App Store
straight away. It goes without saying that you should track the conversion changes
after the upload of the optimized product page elements.

33
Chapter 2. Workflow: A/B Testing Timeline

If you stay consistent with your A/B testing activity, the results will speak for
themselves. Nevertheless, your conversion won’t change overnight. It normally takes
1-2 month to see a new trend gain ground.

7. Follow-up Experiments
Duration: ​3-4 months

It’s crucial to turn A/B testing into an essential part of your ASO strategy. The truth is
that app stores are ever-changing systems subject to constant alterations. That’s why
it’s critically important to run follow-up experiments to scale results.

When it comes to app’s conversion, there is always room for improvement and
well-structured approach to A/B testing can help you become an App Store highflyer.
However, as in all things, success requires time, dedication, and persistence.

34
Chapter 2. Workflow: A/B Testing Timeline

KEY TAKEAWAYS

1. When it comes to A/B testing, top companies use research and elaborate
strong hypotheses based on the collected data.

2. Your variations for A/B Testing should always reflect a quality hypothesis.

3. It’s vital to choose a traffic source and appoint a target audience before
launching an experiment.

4. A/B test won’t even bother you if you decide to test with specialized tools
as they distribute audience members automatically landing these users on
two different variations.

5. Negative results prevent us from making changes that could cost us

thousands of unseized installs.

6. It’s crucial to turn A/B testing into an essential part of your ASO strategy
and run follow-up experiments to scale results.

35
CHAPTER 3

Pages to Optimize
in App Stores
1. App’s Product Page Testing
2. Category & Search Pages Testing
3. Creative Sets Testing within Apple Search Ads

36
Chapter 3. Pages to Optimize in App Stores

Pages to Optimize in App Stores

When you get down to ASO optimization, it’s important to acknowledge all options app
store A/B testing offers. You can test different app store elements within:

● Product page;
● Category or Search page;
● Search Ads banner.

It’s hard to tell which of the options mentioned above is the best as different contexts
call for different test types. Let’s dwell on all these experiment options to make it clear
why and when it’s preferable to use each of them.

I. App’s Product Page

App’s product page is the first thing that comes to publishers heads when they think
about store A/B testing. Indeed, your product page is the place where users get all
necessary information about your app in different forms: text, video, and image.

The app store page provides a playground for mobile marketers to experiment with
several elements of a product page. Naturally, different product page elements have
various impact on conversion. You should keep it in mind prioritizing elements
optimization.

Icons
Experiments with icons can bring you up to 26% increase in app page performance. The
maximum improvement we observed was as high as 560%. Nevertheless, mind that
icons matter more in the context of store search results.

Screenshots
On average, A/B testing screenshots can boost your app page performance up to 20%.
You can experiment with background color, copy, landscape/portrait orientation, or
shuffle the same screenshots to see which one should run in the front. The maximum
improvement we observed got up to 650%.

37
Chapter 3. Pages to Optimize in App Stores

Video Preview
On average, tests with video previews improved app pages’ performance by 16%. The
maximum increase we saw was 110%. However, video preview is a tricky thing, in many
cases app store pages were better off without video preview than with one.

Description
A short promo text before ‘Read more’ button should be your priority as only about 2%
of app store visitors actually read full description. At the same time, a short and catchy
promo text can show an average 35% conversion lift (that said, we’ve seen a maximum
increase in 75%), raising flags for knowledgeable ASO specialists.

Product Page A/B testing Peculiarities:

● App store product page experiments give you a chance to play around all
possible elements distilling a pitch-perfect combination.
● Product page A/B testing is a must for mobile publishers which rely on paid
traffic as the majority of their users skip search phase.
● One of product page tests advantages compared to Search experiments is the
fact that normally you need far less traffic to reach statistically significant
results.

II. Category & Search Pages

It’s no secret that if you want to hit organic traffic jackpot, you should pay closer
attention to optimization of your app listing in the App Store Search. Indeed, Apple
claims that 65% of App Store downloads come from search queries. Ignoring this fact is
an act of folly which may cost your app thousands of lost users.

To fend off organic traffic losses, we collected the most important App Store Search
benchmarks and A/B testing tips.

In general, App Store search is one of the first stages of any user acquisition funnels.
The ratio of users that saw your listing and users that tapped ‘Get’ button is normally
referred to as tap-through rate in the App Store Search.

38
Chapter 3. Pages to Optimize in App Stores

App publishers always wonder what is a good tap-through rate in the App Store Search
in order to get their bearings and understand whether the current state of things calls
for immediate improvements. Unfortunately, each category, each app, each case are
highly individual and there is no golden standard to take cues from.

Nevertheless, here are the benchmarks of a tap-through rate in the App Store Search
which will put some clarity into this issue. We analyzed hundreds of tests in different
categories to identify:

● how App Store search positions favor downloads and further exploration
of an app;
● the average conversion of the first 10 positions in the App Store search.

39
Chapter 3. Pages to Optimize in App Stores

It is no wonder that the first position in App Store Search guarantees the best TTR.
Thus, the average TTR for an app which tops search results is 28.38% while minimum
and maximum TTRs are 0.38% and 83.33% respectively. These impressive results turn
the top position into something definitely worth fighting for.

Having the second position predetermines considerable drop in TTR compared to the
top position (6.15% against 28.38%) but still shows better results than the third one
(2.66%). At the same time, the fourth and fifth positions show similar results (1.36%
and 1.22% respectively)

Yet, you shouldn’t feel down even if your app isn’t in top 5 list of App Store search
results. Our study showed that lower position doesn’t equal to disastrously low results.
In fact, the eighth and ninth positions have even better average TTR than the sixth and
seventh ones (1% and 0.98% against 0.92% and 0.80%).

40
Chapter 3. Pages to Optimize in App Stores

PRO TIP
One of the ways to improve your TTR is to run tests in the context of category
and search pages. There are myriads of ways of improving your App Store
listing.

For example, you can experiment with banner-like landscape screenshots. This easy
trick can bring you up to 45% TTR increase in the App Store search. The thing is such
screenshots draw all user attention and turn the listing of your closest competitor into
a blind spot.

41
Chapter 3. Pages to Optimize in App Stores

Social Point successfully applied this gimmick to their game Monster Legends. They
A/B tested App Store search results page with SplitMetrics. As a result, a banner-like
screenshot in a landscape mode reached 15.4% conversion while the conversion rate
of the closest competitor was only 4.7%. So if you rely on organic traffic rather than
paid one, this tip is really worth testing.

A pitch-perfect video app preview may become another game changer. Nailing the
optimal video structure may showcase your app in the best possible way and it’ll
happen right from the App Store search results page thanks to autoplay. Luring, isn’t it?

Nevertheless, when it comes to A/B testing, keep in mind that only smart complex
approach can help you improve your rank in the App Store search. And remember that
it’s very unwise to leave your position in the App Store search to chance.

Optimize every aspect of store listings you can and it won’t take too long to enjoy
impressive results.

III. Creative Sets Testing within

Apple Search Ads
The introduction of Creative Sets gives Apple Search Ads users a great opportunity to
align various creatives to different keywords. On top of that, now app publishers can
test Apple Search Ads assets.

However, it doesn’t mean that Apple’s update made full-scale A/B tests possible within
the App Store itself. It’s really important to understand how Creative Sets can be used
for Apple Search Ads assets testing and what are the limitations of this experimental
method.

42
Chapter 3. Pages to Optimize in App Stores

Basically, a Creative Set is an assets pack for Apple Search Ads which consists of
screenshots and app previews chosen from your app’s product page in the Store. The
primary goal of these Creative Sets is to add more variation to Apple Search Ads
aligning various assets to ad group keywords and audience segments.

The number of Creative Sets within Apple Search Ads is limited to 10. Each set has an
image number limit as well:

● iPhone: 4 or more screenshots/app previews in the portrait mode and 2 or

more assets in the landscape one;
● iPad: 3 or more screenshots/app previews in the portrait mode and 2 or
more assets in the landscape one.

The process of testing Apple Search Ads Assets with Creative Sets is pretty
straightforward: at creating different Creative Sets and aligning them with various Ad
Groups or Campaigns, you are to track their performance (based on conversion and
tap-through rate).

PRO TIP
When creating Creative Sets try to avoid using screenshots/app previews
denoted as “default”. The thing is the default Apple Search ads with the first
creatives from your store product page is already there. It makes more sense
to compare the performance of your default ads with the one of your
Creative Sets.

43
Chapter 3. Pages to Optimize in App Stores

Apple also recommends using the same number of assets for each Creative Set under
testing. Thus, the same scheme of your Creative Sets will make the results of
comparison clearer. Try to use a maximum of 3 portrait or 1 landscape screenshot/app
preview on iPhone; 2 portraits or 1 landscape asset on iPad.

The introduction of Creative Sets induced the speculation about ‘A/B testing within the
App Store itself’. However tempting, it’s not true.
Experimenting with Creative Sets is indeed a great way of empowering Apple Search
Ads and improving audience response to it. Nevertheless, it doesn’t provide tools for
quality A/B testing of product pages within the App Store.

Here is the list of limitations which preclude calling the new Creative Sets feature a tool
for proper A/B testing:

1. Creative Sets allow tests only in the context of Apple Search Ads.

It’s a good method of optimizing Apple Search Ads banners but the results can’t be
treated as the universal truth of user interaction with your app. In fact, the difference in
user behavior is quite substantial.

44
Chapter 3. Pages to Optimize in App Stores

For instance, according to our latest study on TTR benchmarks, App Store visitors now
detect Apple Search Ads way better and it changes their behavior. For instance, the
average tap-through rate of Apple Search Ads banner is about two times lower
compared to the results of genuine top position in the App Store Search (7.11% against
19.29%).

45
Chapter 3. Pages to Optimize in App Stores

2. It’s impossible to test app’s product page.

Experimenting with Creative Sets, one can draw conclusions regarding App Store
search at most. The results of this testing can’t be applied to product pages of apps.
Furthermore, such Apple Search Ads ‘tests’ don’t allow you to experiment with such
extremely important product page elements as an app icon, name, promo text, etc.

Apple Search Ads is based on keywords and match types so its targeting method is
dramatically different from Facebook, Google AdWords and in-app ads. Creative Sets
don’t allow mobile marketers to use external traffic sources.

That’s why one should be extra careful with the takeaways from experimenting with
Creative Sets if they rely on paid traffic. In such cases, fully-fledged optimization of App
Store product page with help of specialized mobile A/B testing platform like
SplitMetrics is a must.

3. The restrictions of screenshots and app previews within Creative Sets.

As it was mentioned above, one can use only screenshots and app previews from their
product page preparing Creative Sets. It imposes serious restrictions on the testing
ability of experimentation with Apple Search Ads Assets.

Thanks to the recent update of screenshots limit, an app product page in the App Store
can contain up to 10 images. This limit seems more than sufficient unless you plan to
test screenshots with help of Apple Search Ads Creative Sets.

We shouldn’t forget that screenshots and app previews within Creative Sets have the
same order as on your product page. It means you can’t play around their order to its
full extent. For example, such fixed order won’t allow you to put the second screenshot
before the first within one of your Creative Sets.

46
Chapter 3. Pages to Optimize in App Stores

47
Chapter 3. Pages to Optimize in App Stores

This Creative Sets limitation also doesn’t allow you to test polar opposite hypotheses
and creative concepts. Indeed, there is barely a publisher that’ll allow their product
page to be packed with differently designed screenshots with various color schemes.

4. Creative Sets are aligned to keywords.

Creative Sets are bound to different keywords which makes results interpretation
tricky. It’s impossible to tell which screenshot/app preview combination outperforms
others in all respects and should take first positions on your product page.

Let’s assume that Creative Sets testing determined different winners for two different
keywords, this result is quite ambiguous. However, in such cases, you can check
Creative Sets performance on the ad group level to grasp average results and identify
the winner.

5. The limited number of storefronts.

For now, ​Apple Search Ads storefronts​ are limited to 13 countries:

● Australia;
● Canada;
● France;
● Germany;
● Italy;
● Japan;
● Mexico;
● New Zealand;
● South Korea;
● Spain;
● Switzerland;
● the United Kingdom;
● the United States.

48
Chapter 3. Pages to Optimize in App Stores

Presumably, this list will keep growing but for now the storefront limitation is a serious
testing obstruction. It’s also impossible to localize your app’s assets via Creative Sets
testing.

6. It’s hard to measure the statistical significance of Creative Sets testing.

Apple Search Ads doesn’t have a built-in calculator of results statistical significance as
testing is not the primary goal of Creative Sets launch. So you are responsible for all
these calculations yourself. In case of specialized A/B testing platforms, all work is done
for you ensuring quality results.

7. The sample size for testing should be calculated manually.

Sample size is another important ingredient of a successful experiment which Apple

Search Ads testing lacks. You should calculate sample size manually as well, otherwise
your tests won’t be valid and trustworthy conclusions will be out of your reach.

8. No insights into user behavior.

Conversion and tap-through rate are the only metrics that Creative Sets testing can
provide. These metrics can barely tell why App Store visitors like or dislike your app
and its assets. To enhance your marketing strategy, you need way more advanced and
exquisite analytics.

49
Chapter 3. Pages to Optimize in App Stores

For instance, when it comes to app preview analytics, Apple Search Ads testing won’t
tell you the average view duration, or how many users installed your app after watching
the app preview, or what episode of the video triggers the majority of installs. In its
turn, SplitMetrics A/B testing platform can provide all these data and more.

That’s why it’s better to opt for specialized A/B testing solutions that provide in-depth
analytics for all product page and search page elements along with the expertise on
running smart experiments.

9. Creative Sets testing low capacity.

Experimenting with Creative Sets within Apple Search Ads doesn’t give that much
room for manoeuvre. The number of tests you can run simultaneously is limited by the
number of existing Creative Sets.

Moreover, it’s quite time-consuming as the change of concept under the test calls for
product page update and Apple review.

10. It’s impossible to run pre-launch tests.

Pre-launch is a vital period which can doom your app or become a cornerstone of its
success. That’s why it’s so important to spare time for testing an app before it is live.
Optimizing apps in pre-launch doesn’t only guarantees you’ll make the most of your
first days in the store, it also helps to validate ideas and create the app people are
willing to install.

As you can see, the word about free App Store A/B testing doesn’t have much ground.
Moreover, Creative Sets testing can’t be qualified as a free method as you still have to
pay for Apple Search Ads.

50
Chapter 3. Pages to Optimize in App Stores

Nevertheless, Creative Sets is a great way of making your ads more relevant, for
instance, if you have a dating app, it makes sense to run ads with screenshots depicting
male and female characters for corresponding gender groups.

What’s for experimenting with Creative Sets, ​SearchAdsHQ created the tool for
convenient assets testing within Apple Search Ads. Nevertheless, when it comes to
comprehensive ASO strategy, Creative Sets can’t be the only method of testing as
you’ll need more scale, flexibility, and analytics.

51
Chapter 3. Pages to Optimize in App Stores

KEY TAKEAWAYS

1. You can test different app store elements within:

● Product page;
● Category or Search page;
● Search Ads banner.

2. App store product page experiments give you a chance to play around all
possible elements distilling a pitch-perfect combination.

3. Product page A/B testing is a must for mobile publishers which rely on
paid traffic as the majority of their users skip search phase.

4. If you want to hit organic traffic jackpot, you should pay closer attention
to optimization of your app listing in the App Store Search as Apple
claims that 65% of downloads come from search queries.

5. The introduction of Creative Sets gives Apple Search Ads users a great
opportunity to align various creatives to different keywords but it
doesn’t mean that Apple’s update made full-scale A/B tests possible
within the App Store itself.

52
CHAPTER 4

Most Influential
Page Elements
for A/B Testing
1. Icon
2. Screenshots
3. App Preview

53
Chapter 4. Most Influential Page Elements for A/B Testing

I. Icon
Considering the fact that human brain processes visual information much faster than
text​, the way apps’ icons look must be a governing factor for ASO. Icons give us the first
impression and therefore they are the gateway to your apps. ​They can also be the most
impactful app store product page elements that help you grow an app organically.

R​emember that the ​App Store and ​Google Play have different requirements and
recommendations for an icon design. Regardless of the OS for which you are creating
the application, a good icon does not just grab the attention of app store visitors
instantly, it also communicates your app’s quality and purpose.

This is why an optimized app icon is of crucial importance for app store optimization.
On average, the app's icon has a 10-25% potential for conversion increase, and in some
cases, ​your app’s conversion may skyrocket​.

How do we make sure we use the app icon that drives results? ​Core elements of a
converting icon are simplicity, lack of excess visual components, ability to stand out
among competitors without losing touch with the conventional principles of your store
category.

The most difficult part of testing mobile app icons is generating variations. For a start,
you can simply review some of the top performing apps across your categories to find
the styles you could try in the design of your mobile app icon. ​But before you start you
should ask yourself:

● does your icon tell the story and sell the unique features of your app?

It’s essential to ensure that the users understand the message behind your icon. For
example, if we look at selfie app icons, we’ll see that the vast majority feature camera,
lens or a lens-like visual.

54
Chapter 4. Most Influential Page Elements for A/B Testing

Music app icons contain musical notes, sound waves or equalizers. The cooking utilities,
aprons, or chef’s hats appear on recipe apps icons respectively.

Each app also has its strong points. Identifying and highlighting them is essential. So
when developing an app store optimization strategy, you should make sure the app
stands out.

PRO TIP
If your brand is already visible in your target market, using it in the icon is a
must. It will give the app credibility and increase the trust of users.

For example, if you search for “racing”, you will notice that the results contain some
icons having a big publisher’s logo in the corner. You can find a few examples below.

55
Chapter 4. Most Influential Page Elements for A/B Testing

● Does your icon stand the competition in search?

Using your target and related keywords you can see your app’s position in search. Look
at its main competitors. Do their icons look the same? What makes yours stand out?
Getting down to optimization, it’s important to remember that icons matter most at the
app discovery stage.

Once store visitors get to the product page, they pay very little attention to the icon.
So the best way to perfect an icon is to run a series of ​category tests on an app store
identifying, which version performs better in the competitive environment.

56
Chapter 4. Most Influential Page Elements for A/B Testing

App Store Icons Requirements

Whatever style or design you choose, you’ll have to follow the technical requirements
posed by Apple or Google to make your app icon rank. Both companies provide detailed
description of what they expect from a good icon from size to overall user experience.
You can access ​Apple’s​ and ​Google’s​ style guides.

App Icon Sizes

Each app is supposed to have a set of small icons for the Home screen and a larger icon
for the App Store itself.

Here’s the App Store icons size table for different Apple devices:

Device or Context Icon Size

iPhone 180px × 180px (60pt × 60pt @3x)

120px × 120px (60pt × 60pt @2x)

iPad Pro 167px × 167px (83.5pt × 83.5pt @2x)

iPad, iPad mini 152px × 152px (76pt × 76pt @2x)

App Store 1024px × 1024px (1024pt × 1024pt

@1x)

App Icon Attributes

All app icons should adhere to the following specifications.

Attribute Value

Format PNG

Color space sRGB or P3 (see ​Color Management​)

Layers Flattened with no transparency

57
Chapter 4. Most Influential Page Elements for A/B Testing

Resolution Varies. See ​Image Size and Resolution

Shape Square with no rounded corners

To publish your Store Listing in Google play store, a high-res icon is required. The
high-res icon does not replace your app's launcher icon, but should be a higher-fidelity,
higher-resolution version that follows these design guidelines:
● Material icons​ used through Android M
● Adaptive icons​ for Android O

Google Play Requirements

● 32-bit PNG (with alpha)
● Dimensions: 512px by 512px
● Maximum file size: 1024KB

Colors and Styles of Icons

Choosing your icon style and color scheme is close to a full-scale branding decision as
app icons actually represent your company in the App Store and serious discrepancies
between the app design and your corporate identity can discourage even the most loyal
customers.

When you create an app, you definitely know who you are creating it for. Age, gender,
location, language, and other characteristics of your prospects impact the app design.
The same rule applies to icons, and the more precise your targeting is the more installs
you can expect.

It seems obvious that the icon style of a kids game will be drastically different from an
icon of an accounting app in terms of colors and composition.

58
Chapter 4. Most Influential Page Elements for A/B Testing

Colors

Like companies that are known by the colors they use in their branding, apps can also
be determined by their colors. Yet, what is the most suitable color for a new app icon?
There is no certain response to this question. However, the examples and the meanings
of colors can guide your decision.

From Facebook and Twitter to PayPal and Visa: ​Blue ​is an iconic color that makes up
thousands of logos. Colour theorists say that blue suggests reliability, trust, honesty,
loyalty and tranquility. That is why blue is so popular as a icon color, especially for
products that are intended to be used globally and need to transcend cultural
interpretations.

Green ​is often associated with money. Green color is mainly used by 2 categories of
apps. The first one are financial services, as we know that green is mainly associated
with money. The second category includes some apps that promote eco friendly
behavior.

Purple ​is considered to be a female color rather than male (that’s why it is so popular
with selfie app designers). Red is bold and energetic, orange is cheerful, yellow is warm,
and green is peaceful, so you can pick red for a sports app, and green for some health
tracker.

59
Chapter 4. Most Influential Page Elements for A/B Testing

However it’s important to be careful and account all possible associations and
interpretations that go with the chosen color (except blue, as you can guess).

Best Practices of Icon Optimization

Even a well-designed icon based on your branding rules can't guarantee that you’ll win
millions of apps in an app store. Only data-driven approach, testing and optimization of
different elements of an icon (colors, backgrounds, graphics, compositions etc.​), allow
you to boost the downloads of your apps and make the most of your icon.

Make sure you test variations that differ from each other significantly. Small changes
like a color shade or a different angle of the graphics are bad candidates for a split test,
usually showing little to no difference in conversion. Here are some basic optimization
tips:

1. Keep it simple

You probably have quite a few ideas of what to put on the icon. Try a variation with the
simplest concept and bring the number of graphical elements to a minimum. Or better,
choose one. Check if less is really more (converting).

PRO TIP
1024 × 1024-pixel canvas is a challenge in itself. Try the design out on the
device in multiple contexts and sizes. Make sure the icon looks good against a
variety of backgrounds.

60
Chapter 4. Most Influential Page Elements for A/B Testing

2. Try different colors

Your designer probably knows what color combinations look good. The tough part is to
figure out which of them actually work. Test and see if a soft neutral or a bold contrast
background will highlight the main element of the icon along with increasing the
conversion rate.

3. Let your graphics speak for you

For some reason, many app developers are tempted to put their app’s name or some
text in the icon. Usually, this isn’t a good idea, as you have plenty of space in the app’s
title and description. Test if the graphic language will convert better than extra letters
in the icon.

But some some apps use words or letters in their icon; most are branded, but not all. If
you’re using a single letter and it’s a good (and unique) fit, then the letter will lose its
“wordy” quality and become iconic. However, this is more of an exception than a rule.
Remember that an icon is not the same as a logo and it shouldn’t be forced into the
same context.

4. Logo/No logo?

If you use your company’s logo as an icon, play around and test if another concept will
translate the brand as well as the app’s value proposition to user​s. The main issue is the

61
Chapter 4. Most Influential Page Elements for A/B Testing

fact that an app icon is not a logo. They have some similarities the success criteria are
absolutely different.

Logos are vector images and icons are raster ones, logos don’t have to be square but
they need to look good on letterheads, etc. The approach, the tools, and the job for
these two are absolutely different.

5. Add borders

Try to add borders to your icon to make it look clear and attractive on all backgrounds.
Sometimes, simply adding a border without changing anything else makes the icon
appear more “complete” and eye-catching.

6. Derive your own benchmarks

History repeats itself, so analyse your competitors and do something different.

Emulating the leaders is a safe but a tricky route to go down. Your uniqueness is one of
the keys to success. Start with the colors as they do make the difference but don’t
neglect the research not to end up in a hole.

62
Chapter 4. Most Influential Page Elements for A/B Testing

PRO TIP
Pick out 10 best app icons in your category and think how to make yours stand
out. Create a couple of alternatives based on your analysis and start testing.

Icons A/B Testing

As mentioned above, it’s a good idea to start with competitors research and determine
the best practices

PRO TIP
If you study the icons of the top games, you’ll notice that the overwhelming
majority use the same pattern: an icon depicts characters with their mouths
open. It may seem silly, but the trick works. You can use this layout to design
one of your variations.

Testing different characters on your icon is another highly recommended path. For
example, MyTona ran a series of icon experiments for their game, Seekers Notes:
Hidden Mystery. They started by testing a human character against a pumpkin head.
The latter performed 9.3% better.

63
Chapter 4. Most Influential Page Elements for A/B Testing

In the follow-up tests, the icon with the pumpkin head beat the serious man in a hat
once again, but performed only 3.5% better this time. The new icon with the smiling
witch smashed the previous two with a 9.1% conversion lift. This experiment once
again proved the efficiency of ‘open mouth’ strategy and the significance of testing
characters.

Remember that devil is in the detail. Even changing of your background color may
result in incredible growth. The app Hobnob experienced a serious conversion fall after
the icon rebranding and opted for A/B experiments to remedy the situation.

A series of split-tests identified the icon that performed 64% better than the rebranded
one. Surprisingly, the only difference between the icons was in gradient background in
the winning variation.

64
Chapter 4. Most Influential Page Elements for A/B Testing

After users find your app page the icon will to a large extent determine their decision to
install the app or leave for good. You should start working to optimize your icon long
before launching it to App Store.

Here are a few things to keep in mind:

1. Meet your audience’s expectations
2. Let the graphics represent your app
3. Follow App Store’s recommendations
4. Avoid text on the icon
5. Experiment with colors
6. Add borders
7. Be original

… And test everything you can to make sure you are moving in right direction.

II. Screenshots
When a user gets on the product page, screenshots tend to catch the best part of
attention due to its visual nature and a large proportion of page space. Thus, app
screenshots are responsible for grabbing the attention of potential users and argue
them into installing an app.

There’s a considerable part of users whose perception of an app is primarily based on

the quality of its screenshots and app reviews. Therefore, your screenshots should
answer the following questions:

● Can your app solve a particular problem of users?;

● Is it an app of high quality?;
● Is it worth spending time and money on the app?

Still, nobody will examine your screenshots to answer these questions if they’re not
engaging enough. To perfect your screenshots you have to run multiple of tests
checking the efficiency of various design tricks.

App Store Screenshot Sizes and Requirements

65
Chapter 4. Most Influential Page Elements for A/B Testing

First things first, your screenshots are to meet the technical requirements if you want
them to look professional and appealing.

The number of screenshots on each store product page is limited to 10 images on the
App Store. Don’t forget that if you add a video to your store page, its preview acts as
the first screenshot. It’s important to remember the ‘Play’ sign in the process of video
preview designing.

Here’s app store screenshots size summary for different Apple devices:
iPad
● 2048 x 2732 pixels for portrait
● 2732 x 2048 pixels for landscape
iPhone
● 1242 x 2208 pixels for portrait
● 2208 x 1242 pixels for landscape
iPhone X
● 1125 x 2436 pixels for portrait
● 2436 x 1125 pixels for landscape
Apple watch
● 312 x 390 pixels (42 mm).

What concerns the file format, the App Store accepts PNG and JPEG files. It’s vital to
upload files of the best possible quality within accepted resolutions, you can check out
iTunes Connect Developer Guide for more details. Following these simple
requirements, you’ll have no problem submitting your app.

App Store Screenshots Orientation (Portrait vs Landscape)

Choosing between portrait and landscape screenshots is a common dilemma app
marketers face. Some publishers end up submitting horizontal screenshots as vertical
ones forgetting about users’ convenience, so it’s not the best option.

Such decisions are highly individual and require data;driven approach A/B testing can
provide.

66
Chapter 4. Most Influential Page Elements for A/B Testing

Furthermore, Apple changed the number of screenshots per product page lately. Now
store product pages can feature up to 10 screens which entails the extension of A/B
testing playground.

These screens limit increase is on a per-device basis which means you can upload sets
of 10 different screenshots for iPhone, iPad, Apple Watch, and Apple TV.

The screenshots allowance was restricted to 5 images before the update. Now 5 extra
screens can be added to complement product pages. However, mind that the changes
did not concern app preview limit, each app page in the store can still feature maximum
of 3 video clips not more than 30 seconds each.

It’s also important to understand that the update doesn’t affect Search results pages or
Apple Search Ads. They’ll continue to depict 3 portrait screenshots or 1 landscape one
providing no app preview is available on your product page.

Apple assures publishers that new extended screenshot sets will help to show potential
users more of an app’s experience. Indeed, marketers get more space to tell about app’s
core features and competitive advantages and lure app store visitors into installing
their apps.

It makes a lot of sense but only in theory. The harsh truth is that not that many users
scrolled to the fifth screenshot before the update let alone making it to the tenth one.

We decided to have a closer look at how users interact with product pages with 5
screenshots. The following study is based on 1,800 A/B tests of store pages with at
least 100 views per each 5-image set.

It’s important to bear in mind that the overwhelming majority of the analyzed tests
featured optimized screenshots. So, publishers with raw and plain screens shouldn’t
expect the same scroll depth and conversion. A/B testing with SplitMetrics is a great
way of pushing your metrics closer to ones of the industry leaders.

Screenshot orientation plays an important role here. We analyzed hundreds of

SplitMetrics tests to discover the following curious numbers. When it comes to a

67
Chapter 4. Most Influential Page Elements for A/B Testing

landscape set of screens, only 15% of users scroll through all 5 images, these metric is
even more humble for portrait screenshots – 11%.

This result is quite surprising as publishers tend to think that vertical screenshots are
more scrollable. However, our statistics states that horizontal screenshots favor better
scroll depth. Furthermore, they can boast a slightly better converting capacity as well.

68
Chapter 4. Most Influential Page Elements for A/B Testing

Providing users got around to the fifth image, they are more likely to install the app if it
was a landscape screenshot set. In this case, the average conversion rate is about 18%.
When it comes to a fully scrolled through portrait sets, the average conversion is 13%.

Thus, we can assume that if you plan to upload up to 10 screenshots and strive for a
better scroll depth and conversion rate, it’s better to opt for horizontal images.
However, it would be wrong to shut the door on portrait sets.

Each app and each target audience have their peculiarities. That’s exactly what makes
App Store optimization so exciting.

PRO TIP
That’s why it’s essential to A/B test different orientations along with other
product page elements despite overall ​vertical screenshots dominance. Who
knows, your app’s conversion might skyrocket thanks to this simple and
straightforward aspect.

App Screenshot Colors and Backgrounds

Testing colors and backgrounds of your screenshots is a neat experiment idea.
Remember that it depends on your app style. Sure, you’re not restricted to the same
color palette used in your app design. However, it’s good to start with it as it’s an easy
yet effective way of delivering your app’s spirit.

69
Chapter 4. Most Influential Page Elements for A/B Testing

If your app is playful, the screenshots should reflect it, for example, via using vibrant
custom colors for every screenshot. Minimalistic nature of an app can be applied to
your app screenshots color layout.

Here’re a few tips concerning colors used in screenshots layouts:

● colors shouldn’t contradict the message of your screenshots;

● colors used in app design are more likely to convey what it’s like to use it;
● less is more: don’t turn your screenshots into a hot mess of various colors;
● colors should resonate with your target audience. It’s especially important
when you localize your product page.

Soundly elaborated background in its turn can help draw users attention and
emphasize app’s usage context clarifying app’s features. It’s also a good idea to
research background images your target audience like the most.

Incipia studied product pages of top 100 free apps in the Apple App Store and got the
following results:

● Custom background turned out to be the most popular one;

● Blurred images in the background are more commonly used than clear ones;
● Colorful backgrounds are popular in such categories as social networking and
shopping;
● Blue and white are the prevalent background colors;
● The overwhelming majority of apps used in-app screenshots on the product
page (N/A in this graph).

70
Chapter 4. Most Influential Page Elements for A/B Testing

Screenshots Caption

A distinct screenshots trend has emerged lately: app publishers prefer to add a short
descriptive text reducing illustrations in size. However, it’s important not to overdo, a
caption shouldn’t exceed two lines. So caption tests are a must.

PRO TIP
Adding caption makes no sense unless it’s easy-to-read, short, and clear. The
truth is app store visitors don’t open full-screen gallery normally, so it’s
important to make fonts bold and readable even from product page thumbnails
of screenshots. Usage of call-to-actions is encouraged, app’s features are to be
emphasized with verbs.

71
Chapter 4. Most Influential Page Elements for A/B Testing

Short action-packed captions were used in optimization of SongPop2 trivia game. For
example, a brief verb-packed caption “LISTEN, GUESS, AND COMPETE” replaced
extended “The addictive trivia game featuring real music”. The app obtained 10%
better CVR as a result.

Incipia found out what type of captions prevail in top apps in their study. The majority
of apps used a plain white or black text. Captions in screenshots of game apps took a
certain shape, box or banner on top or bottom of the image as a rule.

It turned out that keywords don’t make any difference, so it doesn’t really matter
whether you use them in your captions. In general, apps with some sort of caption had
better ranking compared to ones without any text on screenshots.

72
Chapter 4. Most Influential Page Elements for A/B Testing

Screenshots Localization for App Store

Many app publishers don’t realize how dangerous it is to underestimate localization

impact on app’s performance. A/B tests of localized store pages can help you
understand each market better.

PRO TIP
It’s highly important to remember that it’s not enough to merely translate your
screenshots captions. Localization is to happen beyond the text. After all, you’re
adapting to another culture.

Remember that devil is in details when it comes to localization. Uber nails details with
its product pages for different locales. It seems that the screenshots were just
translated but they were truly optimized. On app store versions for the US and Japan,
we see various maps, destinations, currencies, etc.

73
Chapter 4. Most Influential Page Elements for A/B Testing

Localizing screenshots you make your app more appealing for representatives of
different cultures, thus enlarging your customer bases immensely. So, you’d better take
it seriously, it wouldn’t hurt to consult professional translator and ask a native speaker
to double check renewed localized screenshots.

Ideas for Screenshots Tests

It might be hard to come up with ideas for tests at time. So when you’re stuck, it’s a
good idea to draw inspiration for hypotheses from App Store screenshots core
principles listed below.

First screenshots are to encapsulate the essence of your app

First two screenshots have to be as engaging and informative as possible without

hurting the ease of comprehension. App store visitors should get the core message of
your app straight away.

Choose the most popular feature of your app for the first screenshots based on
insights about users’ in-app behavior.

74
Chapter 4. Most Influential Page Elements for A/B Testing

Highlight value and benefits not merely features

In fact, it’s better to prioritize benefits over features. Your app can be super innovative
and well designed, but nobody will download it unless it solves problems of users. Your
screenshots should convey the value of the app and explain how it makes users lives
easier.

Use all available screenshots

Sure, not every app store visitor scrolls through all screenshot, but when they do it’s
sensible to make the most of the store guidelines. Using all screenshots, you
communicate more value and cool features your app can boast multiplying chances of
the app download.

75
Chapter 4. Most Influential Page Elements for A/B Testing

Make use of social proof (testimonials, media mentions, and awards)

If your app got recognition (from being #1 lifestyle app in Honduras to being mentioned
in a top-notch media outlet), don’t be shy to reflect it in your screenshots. It’ll only
make your core message more credible and speed up users’ decision-making.

Showing target audience that your app is already popular has incredible psychological
power which makes it easier to believe that the app is worth installing. Yet, it goes
without saying that it’s hardly a good idea to fake comments and boast imaginary
awards.

76
Chapter 4. Most Influential Page Elements for A/B Testing

Highlight new features after major updates

Your new feature might be a game changer and it makes sense to put it on the first
screenshot after its release. Don’t ignore Apple updates as well, users should know that
your app is compatible with new devices and services.

By the way, it has a potential of increasing your chances of being featured by Apple.
However, once all these updates are no longer hot news, don’t forget to update your
screenshots.

Pay attention to details

There’s no place for carelessness when it comes to screenshots design. Many app
publishers make same old mistakes over and over again. For instance, if you plan to put
a device on you screenshot make sure that status bars should show a full network icon,
full Wi-Fi icon, and full battery icon. Such little things can produce an unfavorable
impression on users, so be as diligent as you can.

77
Chapter 4. Most Influential Page Elements for A/B Testing

Trusting your instincts might be good sometimes, but it makes way more sense to A/B
test all changes before renewing your screenshots on the App Store. At times, results
of experiments contradict surefire hypotheses. Even at getting great conversion
increase after initial testing, don’t stop. After all, there’s always room for improvement.

Bubble Birds 4 managed to get a ​32% conversion lift ​via optimizing screenshots with
SplitMetrics. Old creatives were redesigned according to game category major trends:
short and catchy captions with most appealing characters and custom background
which brings together art overlays, real screenshots, and gameplay elements. These
helped the company reduce user acquisition costs considerably.

Sure, your app’s prosperity doesn’t solely rest on your screenshots looks and
informativity, but it’s reckless to ignore the considerable impact of this product page
element. We live in the world where people tend to ignore extensive texts in favor of
visual information especially when we talk about app stores.

The majority of app store visitors know how screenshots of a good app look from prior
experience. So, equip oneself with tips and get down to polishing your screenshots via
A/B testing.

78
Chapter 4. Most Influential Page Elements for A/B Testing

III. App previews

Video app preview is considered a dark horse of store product pages. Some publishers
are in the mistaken belief that uploading any video will favor conversion rate growth.
The truth is app previews can skyrocket your conversion as well as drive potential
users away.

It shouldn’t come as a surprise that video quality is a major game changer when it
comes to the efficiency of app previews. However, different types of video previews
work for different kind of apps.

How to reduce a gambling factor of placing app preview on your store product page?
A/B testing is the answer. Optimize your video assets properly, make sure they are able
to boost conversions and only then update your App Store page.

Below you can find recommendations which can help you brush up your app previews
for further A/B testing.

Don’t duplicate images and captions from screenshots

You app previews will show better results if they do not duplicate images and captions
used in screenshots of the product page.This easy yet effective tip reminds us to use
App Store visitors attention span wisely.

For instance, we’ve seen tests where variations with app preview captions duplicating
all screenshot captions showed conversion decrease of about 5%. Thus, when it comes
to your product page, every second of users attention matters so try to engage the
audience without repeating yourself and cover different aspects of the app.

Keep videos up to 20 seconds long

The length of the best-performing app previews is normally up to 20 seconds. This
unsaid time limit encourages you to create clear and straight to the point videos that do
not contain small, hard-to-read and fast-moving texts and avoid screenshots
duplication.

79
Chapter 4. Most Influential Page Elements for A/B Testing

FFStudio experimented with app

previews for their game Bombastic
Brothers. The 15-second video
became the winner showing 16,9%
conversion uplift. All other variations
exceeded the 20-second limit and had
less impressive results. For example,
the conversion improvement of the
runner-up variation was more than two
times worse compared to the winning
one.

Lean toward 2 short videos

If a publisher wants to uncover different sides of one application within their app
preview, it makes sense to split features into several app previews 15-second each. In
the majority 2 short app previews focusing on 2 different features win performing
better than a couple of 30 seconds videos.

Avoid using all 3 videos

Store product pages with 3 app
previews hardly ever win. We’ve seen
3 app previews outperforming 1 and 2
videos only once, but even then users
didn’t watch them until the last
seconds. It once again speaks in favor
of 2 short and eloquent app previews.

For instance, Zimad A/B tested 3

variations and each had one, two and
three app previews per page
respectively. The page with two videos
won with 11,5% conversion rate improvement.

80
Chapter 4. Most Influential Page Elements for A/B Testing

It’s curious that the control page with one app preview showed the second result while
variation with 3 videos showed 9.3% decrease in conversion. It proved the trend we’ve
mentioned above once again.

Mull over your priorities before choosing video orientation

It is better to avoid using landscape app previews on store product pages with portrait
screenshots. In such cases, a video is placed in the Closer Look section which basically
means that it has very little or no influence on CVR. However, it may make sense to use
landscape app previews with portrait screenshots to increase TTR in Search and keep
CVR on the app landing page unchanged.

Thus, you need to sort out your priorities before deciding on the orientation of your
app preview:
● If you rely on organic traffic, it’s definitely worth focusing on increasing
TTR in Search and experimenting with landscape app previews;
● If you focus on paid traffic, avoid orientation mismatch of app previews
and screenshots which puts a video in the Closer Look section.

Word to the partner

Sylvain Gauchet from ​Apptamin​, the agency for app

videos, complemented our recommendations list with
5 insights based on the experience of creating app
previews. These tips can help you get bearings and nail
a perfect video preview for your product page.

1. Start with your value proposition

Your app previews should have a strong start focusing

on the value proposition of the app or game without
going into too much detail at the beginning. Your app
preview can slow down a bit after but should still
remain an “overview”.

81
Chapter 4. Most Influential Page Elements for A/B Testing

Just like an ad in a Facebook feed, your App Preview can be skipped with just a scroll in
the search results. And on the Product Page, you want to make a good first impression.
So try to show something that embodies the value proposition of your app, and if
possible something that is visually appealing.

Without going to the extreme, it’s best if this first part of the video doesn’t dive into
details and instead gives a quick glance at the app.

Using easily readable copy can also be a great way to have a strong start. You can find a
good use of copy below. Never make it smaller than this.

2. Choose your poster frame wisely

Don’t forget about a well-chosen poster frame that complements your screenshots set.
It’s true that the videos autoplay diminishes the importance of the poster frame. But it
doesn’t mean you should neglect it.

Try to think about your app preview frames and

screenshots as a whole. The poster frame needs to
be a frame of your video, so you should plan it at the
same time as you plan your video preview.

The poster frame of the first App Preview will be

displayed in the following cases:
– for a very short time before the App Preview starts
playing in the search results on the App Store. But
still enough time for most users to see it (try it
yourself!);
– while the App Preview of the app below or above
(competitors) is playing in the search results. For
example, while the Candy Crush video plays, we see
the Jelly Splash poster frame;
– before the App Preview video is fully visible on
the Product Page;
– in India and China;
– for any users that have disabled autoplay in the
settings.

82
Chapter 4. Most Influential Page Elements for A/B Testing

3. Make all details distinguishable

Mind that a user should be able to perceive mute app videos in the preview mode. So
make sure that all captions you use in your app preview are easy-to-read and all details
are distinguishable.

All videos autoplay in the mute mode so don’t rely on narrative aspect of your audio.
The footage of your app preview should speak for itself.

4. Know Apple guidelines

Know the guidelines and have a backup plan when experimenting in the “grey area”. It
goes without saying that it’s essential to study Apple app preview ​guidelines carefully
before getting down to your videos to avoid any problems in the course of product
page reviewing:

83
Chapter 4. Most Influential Page Elements for A/B Testing

● App preview length should be between 15 and 30 seconds;

● It’s forbidden to use UI outside of your application so use only the footage
captured within the app itself;
● Showing features available within in-app purchases, it’s obligatory to put
an in-app purchase disclaimer;
● Your video is to be appropriate for ages four and older;
● Creating your app preview, it’s highly important to avoid objectionable
content, violence, profanity, etc.;
● It’s necessary to update your app in the Store in order to upload new app
previews or poster frames;
● App Previews in the App Store are device-specific so mind the required
resolutions:

84
Chapter 4. Most Influential Page Elements for A/B Testing

Historically Apple’s guidelines have been pretty strict. It seems that Apple has loosened
up a bit, and browsing the App Store it does not take long to find videos that would
definitely have been rejected before.

Exploring the “border guidelines” can be interesting if it helps you better portray what
your app is all about. But a review by Apple is still a manual process, which comes with
subjectivity. So it’s almost impossible to know for sure what Apple will approve or not.

That’s why if you experiment with something (showing a live-action video behind a text
screen, a short animation on a screen, special effects over your gameplay, etc.), you
should plan it in a way that it won’t “break” your video in case Apple rejects it and you
have to redo it.

Have a backup plan!

5. Make attention-grabbing your major priority

App previews should be aimed at covering the Attention stage of the A.I.D.A.model
(stands for the terms Attention, Interest, Desire, and Action). So make
attention-grabbing your main priority creating your app previews.

Only an intelligent approach brings significant results. It also may be sped up with
experts assistance. That’s why it’s worth investing in professional video production and
high-quality A/B testing tool.

Getting down to A/B testing, keep in mind that app previews don’t ensure a higher
conversion on the app store page by default that’s why A/B testing is a must. However,
app preview done right is definitely worth the investment.

85
Chapter 4. Most Influential Page Elements for A/B Testing

KEY TAKEAWAYS

1. Icons, screenshots, and app previews are normally the most influential
product page elements so it makes sense to start your A/B testing journey
with them.

2. Core elements of a converting icon are simplicity, lack of excess visual

components, ability to stand out among competitors without losing touch with
the conventional principles of your store category.

3. If your brand is already visible in your target market, using it in the icon is a
must. It will give the app credibility and increase the trust of users.

4. Pick out 10 best app icons in your category and think how to make yours stand
out. Create a couple of alternatives based on your analysis and start testing.

5. When a user gets on the product page, screenshots tend to catch the best part
of attention due to its visual nature and a large proportion of page space.

6. Screenshot orientation plays an important role. We analyzed hundreds of

SplitMetrics tests to discover that when it comes to a landscape set of screens,
only 15% of users scroll through all 5 images, these metric is even more
humble for portrait screenshots – 11%.

7. Adding caption makes no sense unless it’s easy-to-read, short, and clear. The
truth is app store visitors don’t open full-screen gallery normally, so it’s
important to make fonts bold and readable even from product page
thumbnails of screenshots.

8. It’s highly important to remember that it’s not enough to merely translate your
screenshots captions. Localization is to happen beyond the text.

9. Use A/B testing to reduce a gambling factor of placing app preview on your
store product page.

86
Chapter 4. Most Influential Page Elements for A/B Testing

10. Don’t forget about a well-chosen poster frame that complements your
screenshots set.

11. Try to think about your app preview frames and screenshots as a whole. The
poster frame needs to be a frame of your video, so you should plan it at the
same time as you plan your video preview.

87
CHAPTER 5

In-App
A/B Testing

1. Why Is In-App A/B testing Crucial for Long-term Product Growth?

2. What Is In-App A/B Testing?
3. Five Steps to a More Powerful In-App A/B Testing Roadmap.
4. Reasons for In-App A/B Testing.

88
Chapter 5. In-App A/B Testing

I. Why Is In-App A/B testing Crucial for

Long-term Product Growth?

It’s not a big revelation that mobile app adoption has exploded over the past five years.
App downloads increased by 62% in just two years. By 2022 there will be a staggering
31.5 billion​ app downloads across the globe.

With the rapid democratization and proliferation of mobile app adoption over the last
few years, it’s no secret that the product management landscape has shifted.

Very unlike web or desktop apps, the mobile app as a product demands constant and
meaningful user engagement that translates to growth and retention — which
ultimately leads to revenue that can have a lasting impact on your business as a whole.

From working with companies that have run experiments in over 1 billion mobile
devices, Apptimize has discovered companies that iterate rapidly grow on average two
times faster than their counterparts that don’t.

Everything matters in a mobile app feature eco system. Each user touchpoint impacts
product growth — every button, feature, user flow, checkout experience, font, and CTA
has a lasting impact.

In the second half of this ebook, we will walk through the ​what and ​why of in-app A/B
testing, and how it can have a lasting impact on your product’s evolution.

89
Chapter 5. In-App A/B Testing

II. What Is In-App A/B Testing?

Gavin Were​, Director of Google Play Solutions breaks down in-app A/B testing in such
a beautiful and simple way:

A/B testing is a controlled experimentation method that compares two or more

variants so a hypothesis may be confirmed or refuted. The test isolates the specific
test variants from the rest of the system to generate reliable results. A/B testing is
most effective when performed in a live environment when the test population is
unaware the test is running.

At Apptimize, we really believe it’s that simple. If you have a product-focused

hypotheses, A/B testing — when implemented and executed correctly in your product
management toolbox — is an incredibly powerful way to validate those assumptions
through a series of actionable experiments.

Let’s start by reviewing a few terms around A/B testing:

Experiments
Every in-app experiment or A/B test will include a control and typically two variants, an
A variant and a B variant. The control will be your the original version of the app while
each variant will represent a change made to the app.

Goal
For every experiment, you will need to identify the goal, which is an e​vent or
combination of events, that measure the success of an experiment. For example, your
goal could be to increase the number of conversions on the subscription page of your
app.

Hypothesis
A hypothesis is your prediction of how the event or goal will be impacted when you
change certain features of the app.

90
Chapter 5. In-App A/B Testing

Population
The users of your app who could be participants in the experiment. Depending on the
type of experiment, you can decide which segment or types of users to include.
Regardless of whether you target particular users or include any users, it’s important to
randomize their selection in in the experiment’s population to avoid bias in the results.

Participant
The user from the experiment’s population who actually participated in the test. The
sample size is the minimum number of participants your experiment would require in
order for your results to be considered statistically significant.

Statistical Significance
95% statistical significance is the scientific standard. This means that there is a 95%
chance that the actual conversion rate for your variant is better than the actual
conversion rate for your control. Since statistical significance can change over time
based, experiments should run for the number of days it takes for the minimum sample
size to be reached and statistical significance level to stabilize.

Lift
After you run your experiment, you will use the results to measure how the variant
performed compared to the control or original. The lift is the percentage difference in
conversion rate between your control version and the variants.

III. Five Steps to In-App A/B Testing

1. Identify the Hypothesis You Want to Test

Just as within the realm of science, a good hypothesis is basically summed up as an

educated guess. In other words, if you conduct an experiment, what is your expected
outcome? If you change the color of your CTA button, will it drive more conversions, or
repel users from engagement? As a product manager, these are the types of questions
you are aiming to answer. In other words, what levers can you pull to ensure that you
are growing engagement and retention across your mobile app?

91
Chapter 5. In-App A/B Testing

PRO TIP
Your most active users will usually indicate the areas they would like to see
improve. You can mine customer feedback, from both a quantitative and
qualitative standpoint, to discover features to test.

2. Integrate an A/B Testing Platform

While there are several in-app A/B testing platforms on the market, not all are created
equal. You’ll need to find an A/B testing tool that easily integrates with your product
and user segmentation metrics, while enabling scalability as the needs of your mobile
app product become more demanding. For instance, if your team has varying degrees
of mobile development knowledge, a quality A/B testing platform will allow you to
conduct experiments — integrated with powerful third-party mobile analytics tools like
Amplitude, Firebase, and more — to track real-time events without writing a single,
additional line of code.

3. Run an Experiment

This is where the you take all of your hunches and feed them into your in-app testing
platform to validate their impact on your product. Let's look at a simple example of a
new navigation feature A/B test:

Hypothesis: ​Moving in-app navigation drawer to the bottom of the screen will increase
conversions and engagement.

Test A: Current navigation positioned as a drop-down menu at the top of the user’s
screen

Test B:​​ Feature update has moved the navigation drawer to the bottom of the screen

4. Draw Conclusions

Now it’s time to look at the data to draw some meaningful conclusions about your
hypothesis. With a powerful A/B testing tool, you will be able to see how that new

92
Chapter 5. In-App A/B Testing

feature impacted your actual users in real time. It’s easy to view negative conclusions as
failure. At Apptimize, we view these negative results as just important as success. Why?

Because you have eliminated the wrong assumptions, and now you’re further along on
the path to driving more in-app engagement.

5. Take Action

At the end of the day, data does not lie. Whatever the result of your A/B test will end
every debate about new features and updates. As you begin to mine valuable insights
from your in-app A/B testing platform, you will start your journey to adding
user-validated value to your product roadmap.

The results may validate your assumptions that the feature being tested should
become permanent. In that case, the next steps would be presenting the findings of the
experiment to put the feature on the product roadmap. In some scenarios, it might
mean expanding the experiment to include other segments before drawing a
conclusion.

IV. Why In-App A/B Testing?

Stand Out in a Crowded Marketplace

The obvious upside of the rapid, mobile app adoption phenomenon is that the market is
massive. By 2020, mobile apps are expected to ​generate almost $189 billion via app
stores, and in-app revenue generating tactics (native advertising, in-app purchases,
etc).

The challenge of this mobile revolution is that the marketplace for that $189 billion
revenue opportunity is crowded. As of 2018 there are over ​2 million apps in the iOS
app store, and more than ​3 million in the Google Play store. On top of that, the average
smartphone user has more than 100 mobile apps installed on their phone at one time.

That same user engages with 9 of those apps every day, and an average of 30 apps each
month, according to ​TechCrunch​. Not to mention that your ideal-fit users are likely
testing your competitor’s apps while they are testing your product on the same device.
93
Chapter 5. In-App A/B Testing

Capitalizing on your next, big market opportunity requires that you meet the needs of
your users by custom-tailoring experiences for each audience segment (more on this
later).

Practically, this means delivering the right mobile app experiences to the right users at
precisely the right time.

Boost User Retention

As is often the case, user growth is only one piece of the puzzle. Afterall, if you’re
boasting massive user growth and none of your users are regularly engaged, are your
new features you spent months planning and iterating on having a real business
impact?

User retention is among the most common problems mobile apps are tasked with
solving. According to ​Localytics​, 70% of new app users churn in 90 days. What’s more is
that their 2017 industry-wide research indicated that the global average retention rate
was a mere 20%. From an A/B testing perspective, you can make a direct impact on this
KPI by testing how and when you engage with your mobile app users.

Research shows that the majority of mobile app users go dormant within the first week
of using your product. This is a perfect time to start testing onboarding flows, “AHA
Moments”, and other engagement tactics.

Drive a Culture of Data-Driven Experimentation

The principle is simple: if you equip your team members with powerful data to make
smarter product development choices, it will infect your culture. In other words, your
team will adopt an always-be-experimenting mindset. Executing on this principle over
time is a lot more complex.

It all starts with access. According to a recent TechCrunch article on the building blocks
of building a data-driven culture, Warby Parker’s former Head of Analytics, Carl
Anderson says, “Data-driven organizations tend to be very inclusive and provide access
wherever the data can help. (...) It means assessing the needs of individuals, not just the
analysts and key decision makers, but across the whole organization, out to the
front-line of operations.”

94
Chapter 5. In-App A/B Testing

KEY TAKEAWAYS

1. The mobile app growth and revenue opportunity is massive, but the
market is crowded.

2. In-app A/B testing allows you to deliver amazing in-app experiences to

the right users at precisely the right time.

3. Amplify in-app user engagement by A/B testing key points of your user
experience to serve your precious audience the best experience
possible.

4. A/B testing can help product teams determine which features are
priority and the results provide a means for them to get the rest of the
organization onboard.

96
CHAPTER 6

In-App Testing and

Optimization Best
Practices
1. Onboarding
2. Mobile-First Experience
3. Call to Action (CTA) Testing
4. Increasing Subscriptions
5. Monetization
6. Retention
7. Feature Discovery
8. Checkout Flow

97
Chapter 6. In-App Testing and Optimization Best Practices

In the mobile industry, optimization is king.

The mobile app market is one of constant change — methodologies, technologies, and
user preferences are all in constant flux. Most fluid of all? Rather than mapping out the
perfect product, the top apps are always striving for improvement and staying nimble
enough to recognize shifts in consumer demands to change and adapt accordingly.
How do they do this? Through optimization.

It’s important to remember that optimization isn’t only about investing heavily in
building the slickest feature sets and functionality components. It’s often about getting
the most out of what you already have. That’s why the process of mobile app
optimization is so cost-effective — it focuses on tweaking your already-in-place
processes.

In other words, you don’t need to spend huge amounts of time and money. Simply use
what you learn from user data to improve your product. To help you get started, we’ll
run through some experimentation ideas as it relates to different use cases.

I. Onboarding
Depending on who you ask, onboarding is the most critical part of your app users’
experience. Unfortunately, there is no one-size-fits-all approach to onboarding, so
you’ll need to experiment with varying approaches to getting your users up to speed as
quickly as possible.

Having a good onboarding flow makes it easy for your user to start using the app and
helps increase usage, thereby reducing likelihood of churn.

Onboarding can involve the following:

● Welcoming the user and showing them how to get started either by registering,
logging into their account, or setting up their preferences
● Educating the user on how the app works and key features to pay attention to
● Recommending the easiest ways for them to start engaging

98
Chapter 6. In-App Testing and Optimization Best Practices

Let’s take a look at one experiment around the onboarding flow for an app for a
telecom provider.

Virgin Mobile UAE’s Sign Up Experiment

Virgin Mobile launched the first-ever telecom service in the United Arab Emirates in
2017. This was met with a great deal of excitement and complexity, as Virgin Mobile
UAE was only the second brand Under Emirates Integrated Telecommunications
Company (EITC), which essentially meant that they had to leverage economies of scale
in order deliver value to mobile users in Dubai.

The challenge for Virgin Mobile UAE was just as unique as the city of Dubai. For
example, users could use the mobile app to choose their number, customize their plan,
and get their SIM card delivered to their location in an hour or less.

The unique challenge was around user-flow and conversion. For instance, Dubai-based
users spend fortunes on license plates for their cars, and auctions for cool phone
numbers for millions of dollars. Virgin’s initial assumption is that this would make the
choose your own number feature highly valuable. However, the number selection step
was having the highest drop off in the flow by a large margin. The team looked to
testing to better understand their users’ preferences.

Virgin had the idea to

randomize number
assignments to their users,
with an option to change later
as needed. This allowed space
for indifferent users to not be
affected by choice paralysis
and would be more likely to
convert.

The prediction was that at

least 5% more users would
continue to the next step:
registration.

99
Chapter 6. In-App Testing and Optimization Best Practices

In the test, Virgin asked the control group’s users to select between choosing a new
number or transferring their existing number.

The variant had both options at the bottom, but the most prominent element of the
screen was showing them a random number assigned for them, with a big CTA to move
them to the next screen.

The result is that the variant group with the random number got 9% more users to the
registration step. When we later compared the week before the test to the one after,
the drop off rate had gone down by almost 18%.

Here are three best practices to note when onboarding new users:

1. Pre-populating Where Possible

By suggesting a mobile number, Virgin Mobile reduced friction for the user as
they signed up. A user may not be ready to commit to a username, number or
settings. Offer to do the creative heavy-lifting for them and let them know it can
be changed later. This will give them the chance to explore the app in the current
session.

2. Highlight the Value of App

Even if you have a customer that is active on another channel, don’t assume they
know the best ways to leverage the mobile app. Provide some reasons as to how
the app will simplify or enrich their current ways of doing something, from
communicating with their close ones or booking their next hotel stay. For this
part, it’s better to follow the K.I.S.S. principle and keeping it short and simple. Try
to only illustrate one concept per screen.

3. Guided Walkthrough

You may not require much walkthrough if your app has a truly intuitive Ul.
However, guiding your app users through the main features and functionality of
the app can accelerate their adoption. You may have unique features specific to
your app that you also want to showcase.

100
Chapter 6. In-App Testing and Optimization Best Practices

III. Call to Action (CTA) Testing

Not all in-app A/B tests have to be complex, multi-screen product offering tests. In
some cases, small, incremental UX updates can have a lasting and measurable impact.

One example of this is your Call to Action (CTA). While it may not seem to carry the
same weight, it can result in big revenue and/or gains in engagement — and there are a
wide range of CTA elements to test.

You can test everything from the CTA button color, font style and weight to CTA copy,
placement on the screen, the size of the button, and more. Remember, don’t be afraid
to be delighted by testing elements that don’t follow convention, common sense, or
dare we say, brand. ​Here are three best practices to note when testing CTAs:

1. These Are Not The CTAs You’re Looking For

It’s not just what you’re changing or adding, but sometimes it’s about what the
user doesn’t see that helps boost conversions. Consider minimizing distractions
to help the user focus on the primary CTA. Try hiding buttons or links that may
clutter the screen.

2. Context Matters
If you present an “Upgrade Now” CTA without articulating the benefits of the
more expensive plan, it would be easy to see why conversion would suffer. Try to
make the benefit more descriptive and personalized. For example, if you were
doing a test for a media app, go beyond stating that the user will “receive
unlimited access to articles” upon upgrading. Instead, you could test more
detailed text that lets them know that they will “get unlimited access to over
10,000 travel articles”.

3. Go Bigger
A simple yet effective test could be increasing the size of the button or CTA text.
When dealing with limited screen size, try experimenting with different widths.

102
Chapter 6. In-App Testing and Optimization Best Practices

III. Increasing Subscriptions

Anyone who is in the product management business for mobile apps loses sleep over
subscription optimization. This is especially true for product managers who are
managing apps that must be monetized through subscriptions. While there are a lot of
important KPIs to concern yourself with, optimizing subscription growth is likely top of
mind most of the time.

Take the GPS-enabled running tracker Runtastic, for example. They launched a
Dynamic Variable A/B test for their Android and iOS apps. The original variant featured
a promotional statement, while the test variant included a real user review along with a
5-star rating.

The user review variant produced a 44% increase in paid subscriptions for Android
users and no significant change for iOS users. Based on the results of this test,
Runtastic only updated their Android paywall screen.

103
Chapter 6. In-App Testing and Optimization Best Practices

To increase purchase conversion rate, he wanted to test two strategies:

1. Increasing paywall views through tactics such as content locking and signposting
2. Improving payment funnel through tactics like discounting and product
marketing

To influence Average Order Value, he would attempt the following approaches:

1. Shifting product mix through making longer plans the default

2. Increasing prices using price elasticity testing

Test 1: Content Locking

The team tested restricting access to all languages to Premium users only.

Hypothesis
Hardcore free power users who learn several languages at a time would upgrade to
continue learning.

Result
They experienced 83% conversion rate increase for the power user participants
(around 20% of our user base learned >1 language) – one of the strongest they’ve seen
through hundreds of experiments!

Test 2: Discounting
Busuu started experimenting with offering 50% discounts on 12-months plans only
instead of 30%.

Hypothesis
They have virtually 0 cost per marginal unit sold – they could literally discount to 90%
so they gave it a try.

105
Chapter 6. In-App Testing and Optimization Best Practices

Result
They saw a 40% increase in conversion and
88% increase in bookings, as users went for
longer plans. This is because an increased
discount made longer plans look more
affordable.

Test 3: Price Testing

One experiment they ran focused on incentivising users to choose the yearly plan by
breaking down the yearly price.

Hypothesis
The yearly price had already been broken down to its monthly equivalent a couple of
years ago – that is, instead of showing $60/year, they show $5/month above the total, a
very common practice to anchor customers to the lower price and make yearly plans
comparatively more attractive. Just from this, they saw a 12% increase in purchase of
annual plans.

Lesson Learned
The more you break down prices to more granular time scales, the more the “savings”
from longer to shorter plans appear to be smaller, e.g. 5£/month on yearly plan or
10£/month on a monthly plan VS 0.16£/day on yearly plan vs 0.33£/day on monthly
plan.

106
Chapter 6. In-App Testing and Optimization Best Practices

As the savings difference appears much smaller, it changes the trade off between
savings and flexibility of cancellation and pushes users to go for shorter plans, reducing
overall revenue. Make sure you test your price breakdown!

Test 4: Help, Suspension & Discount

Antoine’s team also decreased cancellations by testing multi-step flows that addressed
possible reasons for cancellation.

Hypothesis
Prevent users from cancelling by providing a multi-step flow with multiple winback
tactics

Test Flow
As users go through the cancellation flow, they are presented with:

● A help screen which inquires whether the user might just need some help with
the service
● If the user goes ahead with the cancellation, another screen offering them a
discount if they stay
● If the user goes ahead with the cancellation, another screen asking them about
the reason for their cancellation (from which Busuu gathered really good data on
churn reasons). Then the flow branches:

○ If the user chooses “the free option is enough for me” they are presented
with one last screen reminding them of the benefits of a premium
membership and what they will lose if they cancel
○ If the user chooses “I want to change my subscription” – busuu realized
some users cancelled because they wanted to switch plans to longer ones,
so they presented them with the option to do so
○ If the user chooses “I am too busy” busuu offers them the ability to pause
the subscription
○ If the user still continues with the cancellation, busuu finally presents
them with a confirmation screen and a button, which if clicked, will end
their subscription

Results
Through this flow they saved nearly 30% of cancellations from happening.

107
Chapter 6. In-App Testing and Optimization Best Practices

V. Retention
User retention is an ever-moving target. A study by Appcues indicates that mobile apps
only have 21% retention on day one, and that quickly dips off by the end of the first
week. In your users’ early stages, it’s critical to show your app’s inherent value early and
often.

This is best achieved by helping your audience develop habits within your app’s
experience flow right out of the gate.

Implement a surprise and delight strategy by engaging with new users right where they
live—on their mobile devices and in relevant communities for educational and
engagement purposes.

Check out chapter 8 for more information on tried and true mobile retention
strategies.

VI. Feature Discovery

Regardless of the problem your app is trying to solve, new users downloaded your app
to meet a very specific need. A good feature discovery flow within your app gets your
assumptions out of your users’ way, and lets them accomplish the task at hand as
quickly as possible.

How the Social Media Triumvirate — Facebook, Twitter and LinkedIn — Use Tool Tips
for Feature Discovery

In-App Messaging at Facebook

108
Chapter 6. In-App Testing and Optimization Best Practices

Twitter’s GIF Helpers

VII. Checkout Flow

In the retail and eCommerce worlds, the smartphone checkout experience can be big
business. According to ​Recode​, 2017’s Black Friday saw almost $3 billion in overall
purchases, with 27% of sales and almost half of all traffic came from mobile phones.

Simply put, it matters how seamless it is for users to move through your retail app’s
checkout experience.

109
Chapter 6. In-App Testing and Optimization Best Practices

Though most people probably don’t know what an “SSL/TLS encrypted payment” is
without looking it up, Vinayak Ranade, Director of Engineering for Mobile at Kayak
noted​ that “when we removed the messaging, people tended to book less.”

Letting people know that their information is safe with your app can boost buyer
confidence and app sales by eliminating any doubts from the user’s mind.

111
Chapter 6. In-App Testing and Optimization Best Practices

KEY TAKEAWAYS

1. Experimentation requires some boldness to try and test new,

unorthodox UX ideas. In other words, don’t be afraid to experiment with
onboarding methods that are a little strange and counter intuitive. You
may be pleasantly surprised to see a significant uptick in conversions.

2. When in doubt, get of your user’s way. The job of your app’s UI is to
make it second nature for your customers to navigate and gain the most
value possible out of every feature and in-app experience. The catch is
that you will need to experiment constantly to evolve to a friction-free
mobile app.

3. If you aren’t focused on delivering personalized experiences for your

users, you are no longer competitive.

112
CHAPTER 7

Building
vs.
Buying Your A/B
Testing Solution

1. Overview
2. Internal Resources
3. Accessibility
4. Cost
5. Innovation
6. Security

113
Chapter 7. Building vs. Buying Your A/B Testing Solution

I. Overview
At this point, you probably have already come to the conclusion that in A/B testing and
experimentation is a winning strategy for creating amazing mobile user experiences.
You can use this powerful user data to create user flows that directly impact revenue,
retention and engagement growth. When it comes to sourcing your in-app A/B
experimentation software solution, it all boils down to one question: do you build it or
do you buy the solution?

There are pros and cons to having an in-house or third-party Software-as-a-Service

(SaaS) solution. In this section, we will simply breakdown some essentials to consider
when answering the build-or-buy question.

II. Resources
Do you have the internal engineering resources to build a custom software solution?
This is an important question to answer. You may have internal stakeholders that make
the case that tech giants like Google and Netflix have their own internal customization
tools. The only problem with that assertion is that they are Google and Netflix. They
have no shortage of star power at their disposal. In fact, few brands have “extra”
engineering talent to throw at a custom, internal A/B experimentation tool.

Most organizations are simply hiring new engineering talent to keep up with growth
demands placed on their product. In fact, in most cases making the simplest changes to
a product can result in late nights on the development side of your organization.

To truly scale and grow an in-app A/B testing program, most organizations must rely on
dedicated, third-party tools to achieve their business goals.

III. Accessibility
Perhaps the biggest problem with in-house software is that not everyone within the
organization has access. A/B testing at its best a democratic, transparent process
where multiple teams have access to the same data to make informed product
decisions.
114
Chapter 7. Building vs. Buying Your A/B Testing Solution

For example, without building in an intuitive, plain-English results dashboard, if

someone on your Marketing team wants to analyze the impact of an A/B test, they may
need to understand in-depth statistics to evaluate the results well. Many home grown
A/B testing solutions don’t have a dashboard to display results, so someone in analytics
has to write SQL to pull data, posing a barrier between the end user and impactful data.

Without a system optimized for efficiency, product managers will be limited in their
efficiency and velocity — whether it’s launching an experiment or interpreting the
results. Given that the results of an experiment have meaningful impacts on business
KPIs, this technical constraint can be a
serious roadblock in your growth.

On the surface, building your own experimentation tool might seem like the right
choice. But when you zoom in on an organizational level, it can slow the agile scalability
work flows that many early-to-middle stage technology companies need to hit all of
their business goals.

IV. Cost
Unfortunately, the cost of building your own experimentation platform is not just
limited to upfront development costs. If that were the case, it would make a lot more
sense for more smaller organizations to build their own.

The more costly portion of the “build-your-own-tool” equation comes down to

maintenance. In fact, around ¾ of your total cost of ownership (TCO) is related to
maintenance. This includes everything from managing servers to regulatory data
compliance to managing and fixing bugs once the software is deployed on your own
servers.

You incur all of those future costs when you ship a "light-weight" custom A/B testing
framework, because the implicit expectation is that you will fix, adapt and evolve the
framework as the business grows. Beyond the considerations listed above,
organizations also need to consider the cost of infrastructure (included in the costs of
investing in a SaaS system).

115
Chapter 7. Building vs. Buying Your A/B Testing Solution

V. Innovation
Lastly, you need to consider just how quickly you need to innovate and iterate on your
own product. Even if you have the extra engineering talent and budget to allocate to
building your own platform, you still need to consider the long term effects of slowing
down your innovation process. Your rate of learning and innovation is determined by
how quickly you can experiment. If every experiment you run requires an engineer to
write some custom code, you will likely run fewer experiments.

We have found that in-house solutions are often built for a specific use case and are
ultimately limiting as the experimentation roadmap develops. Without years of
knowledge built up through practice and observation, it's hard to come up with an end
product that will fit nearly all possible edge cases.

VI. Security
Software security and reliability seem like no-brainers when building your own in-app
experimentation platform, but they are also the most difficult and expensive to execute
on. Everyone knows that their in-house applications need to handle user data securely
and effectively. But do you have the internal engineering resources to dedicate to data
security? According to Michelle Drolet, trusted data security veteran, 58 data records
are stolen every second — that amounts to about 5 million stolen data records a day.

It goes without saying that there is significant financial risk with less-than-secure
applications — especially those used for internal purposes. The Poneman Institute’s
2017 Data Breach Study reveals that the average cost of a stolen data record is about
$141. In the United States alone, the average cost of a single data breach is over $7
million.

In other words, it doesn’t take a rocket scientist to put together that data security is
essential for any application. Beyond the threat of lost data is the actual, day-to-day
cost of developing and maintaining in-app security measures across your internal
experimentation platform. This means that in addition to your “extra” engineering
talent you’ll need to build the actual software, will likely need a Data Security or a
Security Intelligence engineer to ensure that all internal user data, as well as external
user data (used for experimentation purposes) is secure.

116
CHAPTER 8

Mobile
Retention 101
1. Gather User Feedback Early and Often
2. Identify and Eliminate Friction
3. Build Better Usability Habits
4. Test Every Release
5. Repeat

117
Chapter 8. Mobile Retention 101

I. Gather User Feedback Early and Often

Unless there is something inherently broken inside your app, the root of your churn
issues is likely rooted in the disconnect between what you think your users need, and
what they actually want.

Poor UI decisions, CTA typos, and distracting pop-ups could all be relatively small
problems with quick and easy fixes that could improve retention.

This is why it’s essential to have a mechanism for collecting and acting on customer
feedback at every possible touchpoint within your app. Luckily, there are many tools
out there that make it quick and easy to gather actionable user insights in real time.
Here are some of the best customer feedback solutions that you can integrate on the
backend of your app:

Apptentive

AskNicely

Delighted.io

GetFeedback

SurveyMonkey

II. Identify and Eliminate Friction

Friction is mobile retention’s worst enemy. Even if you have the functions of the app
are amazing, everything can fall flat for users if the UX is terrible. Terrible UX is at the
root of all friction within your app, and friction is anything makes it harder for users to
move from point A to point B in your app with ease.

HotelTonight is the leading mobile-only solution for last minute hotel bookings, with
over 20 million app downloads across 1,700 markets in 36 countries. While they are far
ahead of their competition when it comes to solving critical UX challenges for their
customers, they wanted to streamline the checkout flow further to increase
conversions.
118
Chapter 8. Mobile Retention 101

Their hypothesis? Eliminate the mandatory account-creation screen during checkout to

increase bookings — reducing friction.

The test was fairly simple. HotelTonight used ​Apptimize Dynamic Variables to test the
elimination of the account setup screen from the checkout flow. The control in the
experiment had the usual mandatory account setup screen. The variant simply asked
users for a name and email on the Order Summary screen without a mandatory
password creation field.

By removing superfluous UX elements, HotelTonight increased bookings by 15%.

Amanda Richardson, HotelTonight’s VP of Product details the experience: “Instead of
only focusing on what you can add, you should also focus on testing what you can
remove. Sometimes the best way to convert is to just eliminate.”

Identifying and eliminating friction takes some detective work. For example, say you
have an e-commerce mobile app. Currently, users have to create an account to make a
purchase. Your product team hypothesizes that allowing users to checkout as a guest
might drive more conversions.

In most cases, this involves performing many A/B tests to ensure your are night hiding
your app’s features, but building habit-forming experiences that aid your users in taking
advantage of your high-value app features.

III. Build Better Usability Habits

Every product manager wants users to visit their app habitually. Unfortunately, habits
aren’t formed overnight. Making every app interaction habitual starts and ends with
engagement. This means that your app has to make an appearance every users’
day-to-day activities. .

We like to call this the surprise and delight strategy. By engaging with your users early
and often you can create a sense of anticipation of your engagements with them at
every touch point. Here are three major ways to start building better app usability
habits with your customers:

119
Chapter 8. Mobile Retention 101

● Push. Push should be used as an extension of your app. It extends your app’s
utility outside of your UI.
● Text. Texting is very personal, so you have to be careful not to use it needlessly.
It is great to send time-sensitive updates such as balances or appointment
confirmations, but not to shamelessly promote your product.
● Social. Engaging in a community is the best way to spread awareness about your
app and subtly remind users of your product every day. Engage users
consistently in places where they already hang out. Then a request to check out
new feature won’t seem so jarring.

IV. Test Every Release

To build for their users, product managers attempt to think from their users’
perspective. Because of their proximity to the product, however, that usually proves
impossible. The whole product team has to operate under the assumption that their
product is useful—and a filter of skepticism is near-impossible to superimpose on top of
that.

That’s why you need to bring real user input into your development process to get
validation on the features you’re building. Use Apptimize’s feature flags to validate
something as basic as the idea behind a feature, or a nuance as small as the type of font
you use.

For example, say you have an e-commerce app like Target’s. You’ve just launched a new
store locator feature, and placed it at the top of your mobile app. instead of launching
the store locator feature to all your mobile users, you could use a feature flag to roll out
the store locator only to users who live within 10 miles of a Target store. That way, you
can test the feature on the users most likely to use it, before rolling it out to everyone
else.

V. Make Retention Repeatable

The goal for all mobile apps is to not only get users to stick around, but to grow in value
over time. That means you want your users upgrade, or become an evangelist for your
app. In both cases, you’ll have users that don’t just benefit from your app—they’ll be
invested in its success. You’ll grow regardless of how much you spend on acquiring new
customers. This is great news for the product manager.

120
Final Thoughts
Thank you for taking time to look through this book. We believe that by following our
recommendations, you can A/B test like a pro — a skill that is becoming more and more
relevant across many disciplines.

App developers claim that app optimization has dramatically changed in recent years.
It is no longer enough to put keywords in an iTunes Keyword Field and use rough and
ready screenshots to optimize an application. Instead, you should complement your
strategy with thoughtful analysis of store elements and in-app actions, reputation
management and consistent testing and optimization.

App developers follow new rules — and there’s no wonder, both new startups and
flourishing enterprises aspire to make the most of their marketing budgets by
optimizing product page elements and in-app flow.

Boosting conversion while spending less and less on Customer Acquisition Costs
(CAC) sounds like a dream. Smart A/B testing can help lower CAC, while maximizing
your application’s earning power. . To maximize conversions across every aspect of
your app (from in-app UX elements to the store page itself)=you should master the art
of mobile A/B testing.

1. Avoid building up new user acquisition channels until you are sure your app
page converts.
2. Start with a mature hypothesis based on customer surveys and competitor
research.
3. Identify elements that cause friction in your acquisition and retention flow. Are
there issues stunting your conversion rate and user growth? Answering these
questions are key to successful experimentation.
4. Know that A/B testing is an ongoing process that should include a successive
series of tests.
5. Choose traffic channels and target audience wisely.

Split-testing not only boosts store and in-app performance on both paid and organic
traffic, it also becomes a source of core product data which should support every rapid
growth strategy. ​Happy A/B testing!

121
Thank you for reading the book!

We'd love if you share your knowledge

with your friends and colleagues!

Learn more about A/B testing

with SplitMetrics at ​SplitMetrics.com