Growth
A/B test
By Rubén García · 3 min
An experiment comparing two or more versions of something (a page, email, ad) to see which performs better.
Traffic is split randomly between versions and one primary metric is measured until statistical significance. Booking.com runs over 1,000 at once.
The key is random assignment: if versions get comparable audiences, the metric difference can be attributed to the change, not chance or other factors.
Before starting you define the hypothesis, primary metric and required sample size. Stopping the test as soon as a version "looks" like it's winning is one of the most common mistakes and produces false positives.
Beyond classic A/B there are multivariate tests (several variables at once) and A/B/n tests (more than two versions).
