Scribd
Upload
61 views2 pages

(Artifacts For Reforge) A - B Test With UI Change in Search Bar at Metro

An A/B test was run for 16 days to test changes to the visibility of the product search bar. The test found a 10.83% improvement in interaction with the search bar and a 5.9524% increase in purchases per user. The changes led to a 4.6231% revenue increase amounting to xxxxxx rubles over 16 days, equivalent to xx.xxx.xxx rubles per year.

Uploaded by

NgocQuyen123
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
61 views2 pages

(Artifacts For Reforge) A - B Test With UI Change in Search Bar at Metro

An A/B test was run for 16 days to test changes to the visibility of the product search bar. The test found a 10.83% improvement in interaction with the search bar and a 5.9524% increase in purchases per user. The changes led to a 4.6231% revenue increase amounting to xxxxxx rubles over 16 days, equivalent to xx.xxx.xxx rubles per year.

Uploaded by

NgocQuyen123
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

A/B test with UI design change in the search bar

Date: 24.03.2024
Idea: a digital experience agency REO
PM: Andreev Andrey
Campaign Type: UX/UI improvement

How long did the test run (its timeframe)?


16 days

What is our hypothesis?


Improving the visibility of the product search bar may contribute to an increase in average revenue per user due to increased search frequency and
more effective interaction with customers.

What are the main concepts, designs, and structural elements?

What methods and tools are we using to measure data and make decisions?
The test was conducted using Gravity Field on client-side browser with a 50/50 split on the website
Decision threshold: .05 level of significance, .80 power
Time window: session, attribution on event trigger
Key success metric: ARPU with MDE +5%

What is the reach of the test? Which specific customer segments saw the variants?
Random user split 50/50 on the website

Limitations:
------
Results:
It was found that users interacted with the search bar 10.83% better than the variant in the control group. A significant increase in purchases per
user was noted, with a growth of 5.9524% with statistical significance (p-value = 0.011) compared to the control group, confirming the
attractiveness of design changes and their stimulating effect on purchases.
What is the business impact?
The variant with the blue frame and shaded magnifying glass brought a significant revenue increase, with a 4.6231% increase compared to the
control group, indicating a positive influence of visual elements on user behavior and purchasing decisions. Over 16 days, the increase amounted to
xxxxxx.xx rubles, which extrapolated to a year equals xx.xxx.xxx rubles.
Variation Users Revenue Revenue/User Revenue Uplift Revenue P2BB Purchases Purchase/User Purchases Uplift Purchases P2BB AOV AOV Uplift AOV P2BB

Control 713212 28374910 69.127 0.0 97.22 5982 0.0145 0.0 1.04 9167.2029 0.0 97.8

Variation 713027 29766049 71.7136 9.2463 63.38 6348 0.0149 9.9524 98.96 9076.4244 -2.0039 69.79

*all numeric data is fake except for relative ones

What risk or confidence level in the current situation or decision?


Risk is minimal, confidence level 9 (ICE Framework)

Conclusions:

As a result of the test, interaction with the search bar improved by 10.83%, number of purchases per user increased by 5.9524% (p-value =
0.011). Visual changes, including the blue frame and shaded magnifying glass, led to a revenue increase of 4.6231%. Over 16 days, an increase of
xxxxxx.xx rubles was obtained, which is equivalent to xx.xxx.xxx rubles per year. UI experiments proved to be effective, emphasizing the importance
of further development in this area.

Recommendations:

Transfer to server side

Next Steps:

1) Switch to 100% traffic 2) Transfer to server side through the product team

Service Information

Contact: For additional questions or clarifications, please contact me on telegram https://t.me/and_and

Asana: link
Colab: https://colab.research.google.com/drive/1E9noCkK41G4IUFcxMI4SMJ3kXcKroycv?usp=sharing

You might also like