Finalize your experiment's statistical preferences

Amplitude Experiment uses default statistical settings for experiment analyses: 

• CUPED toggled off
• Bonferroni Correction toggled on
• Custom Exposure Settings toggled off
• Test Type set to Sequential
• Confidence Level set to 95%

As you review your results, your experiment may benefit from modifying one or more of the default settings. To modify the default statistical preferences, follow the steps below:

1. From the Settings tab, click the Edit icon in the Analysis Settings card.
2. In the fly-out that appears, go to Stats Preferences.
3. Make the desired edits to the default settings, and then click Save.

The Stats Preferences can be modified at any step of an experiment, but they are most beneficial for the final analysis after the experiment has ended.

CUPED

Controlled-experiment using pre-existing data, also known as CUPED, is an optional statistical technique meant to reduce variance in Amplitude Experiment. Toggling CUPED on means that Amplitude Experiment will account for possible varying treatment effects for different user segments. There are situations where CUPED would not be the best choice for your experiment, such as targeting only new users in your test.

The random bucketing process can sometimes deliver unbalanced groups of users to each variant. This is known as pre-exposure bias, and it’s one of the things CUPED is meant to address. If you don’t use CUPED for your experiment, this bias will persist. This is why you may notice differences in the mean-per-variant when running the same experiment with and without CUPED. 

For a more technical explanation, see this detailed blog post.

Read more about CUPED and how it can affect your experiment results in this blog. 

Bonferroni Correction

Amplitude Experiment uses the Bonferroni correction to address potential problems with multiple hypothesis testing. Although a trusted statistical method, there are situations where you may not want to use it when analyzing your experiment results. One might be if you want to compare results with those generated by an internal system that does not support the Bonferroni method. In this case, and if you're willing to accept higher false positive rates, toggle the Bonferroni Correction off.

Custom Exposure Settings

You can allocate a timeframe, or window, to the metric event for it to be considered a true conversion event. The window can be set in seconds, minutes, hours, or days. 

Test Type

You can choose between a sequential test and a T-test when analyzing your experiment. Sequential is usually—but not always—the better choice. For example, you may have too small of a sample size for sequential testing.

Read more about how to utilize a T-test and sequential testing in Amplitude's Help Center.

Confidence Level

The confidence level measures how confident Amplitude Experiment is that it would generate the same results for the experiment if you were to roll it out again and again. The default confidence level of 95% means that 5% of the time, you might interpret the results as statistically significant when they're not. Lowering your experiment’s confidence level will make it more likely that your experiment reaches statistical significance, but the likelihood of a false positive goes up. You should not go below 80%, as the experiment's results may no longer be reliable at this point.