A/B testing with AI: how agents multiply growth experimentation (and what they can't do)
Booking.com runs more than 1,000 concurrent experiments. What Optimizely and Statsig agents already do, a realistic flow for small teams and the statistical mistakes AI won't fix.
By Rubén García · October 7, 2026 · 3 min
An A/B test compares two versions of something (a page, an email, a price) by showing each to a random group of users, to learn which works better from data rather than opinion. In growth, whoever experiments most learns fastest. The bottleneck was never ideas but the time to design, launch and analyze each experiment. That's where AI agents are changing the pace.
The benchmark: Booking.com
Booking.com runs more than 1,000 concurrent experiments at any given moment, according to its own experimentation team. Its model, described in an academic paper by its engineers, is based on democratizing tests: anyone on the team can launch one, not just analysts, always inside a platform that enforces statistical best practices.
1,000+
concurrent experiments at Booking.com
Oct 2025
Optimizely launches variant and review agents
Dec 2025
Statsig adds AI experiment summaries
What agents already do
In October 2025, Optimizely launched an agent that generates test variations automatically and another that reviews experiments and recommends changes to reach statistical significance. Statsig, a platform used by companies like OpenAI, added AI-generated experiment summaries in December 2025 that explain results in plain language.
Your take, in one click
How many A/B tests do you run per month?
A realistic flow for a small team
- An agent proposes hypotheses from your analytics data and user feedback.
- Another generates the variants (headlines, landing pages, emails) following your style guide.
- A person approves the hypothesis and primary metric and launches the test.
- AI monitors that the test doesn't break (errors, sample imbalance).
- When it ends, AI summarizes results and suggests the next experiment.
What AI can't do
- Choose what matters to the business: the team decides the primary metric.
- Fix insufficient traffic: with few visitors, no test reaches significance.
- Replace judgment on a variant's brand fit or ethics.
How it fits your growth strategy
Experimentation is the engine that optimizes each step of a growth loop. It also pairs well with marketing agents for content and ads, as covered in AI agents in marketing.
The rule that doesn't change: one clear hypothesis and one primary metric per experiment. AI speeds up the process; it doesn't replace the method.
Frequently asked questions
What is an A/B test?
An experiment that shows two versions of an element to random user groups to measure which improves a specific metric, such as conversion.
Can AI run A/B tests?
It can generate variants, propose hypotheses, monitor tests and summarize results. Optimizely and Statsig already include these features. Decisions about the metric and the business remain human.
How many experiments does Booking.com run?
According to its team, more than 1,000 concurrent experiments at any given time.
