Optimizely Stats Engine – In 1 Minute

For this week’s One Minute Monday, Robin Pam from our Product Marketing Team team explains the benefits of Optimizely’s Stats Engine, a statistical framework for calculating A/B tests.

Transcript:

So you have your test set up, and you press go. How many visitors do you need before you can be confident in your results?

In the past, our advice was to use Optimizely’s sample size calculator to see exactly how many visitors you would need to run to each variation before making a decision. Now, with our Stats Engine, you can just set the test, and stop it once Optimizely says your results are significant. Stats Engine is designed to eventually detect any effect that exists. The great thing about this is you don’t have to worry about seeing results that won’t hold up over time, no matter when you look at them.

A/B test sample size calculator.

You can still use our sample size calculator to give you a ballpark of how many visitors it might take to see the size difference you want to measure. Calculating a sample size is effectively a reflection of how long you are willing to wait for your test to show significant results. The number of visitors you’ll want to plan for depends on the minimum detectable effect you’re willing to wait for. The smaller the effect you want to see, the longer your test will likely need to run.

We recommend inputting the smallest effect you’re willing to wait for into the sample size calculator. If Stats Engine finds a larger effect, you’ll know about it earlier—in some cases, up to twice as fast. If your test is still inconclusive at that sample size, it might be time to move on to the next idea.

Learn more about the Optimizely Stats Engine: why we made it, how it works, and why it’s better.