Authors: Edgar Cano, Daisy Duursma, Nils Skotara, Melanie Mueller

Experimentation is at the core of product development in Booking.com, powered by our in-house platform, “ET” (Experiment Tool). At any given moment, we run approximately 1,000 parallel experiments to evaluate product changes. These experiments or A/B tests allow teams to directly compare a new version of the website against the existing one, validating hypotheses about how specific changes impact important metrics.
In our organization, these pitfalls became more evident as our experiment volume grew. We observed that experimenters might set an arbitrary “two-week” duration without thinking about sufficient power, or extend a test until results “became significant” or “trended positive.” Knowing that this lack of consistency leads to flawed decision-making we dedicated significant effort to increasing the quality of our experimentation process, making Experimentation Quality a key KPI for our program. However, identifying the problem was only the start; the greater challenge is how to implement these standards across a large organization.
Enforcement vs. Education
When deciding how to scale quality, we faced a fundamental choice: Do we enforce strict controls or we rely on education. Ultimately, we left it to product teams to decide how to conduct their experiments. This choice entailed several trade-offs:
- Enforcement: Ensures comparability, consistency, and reliability. However, it comes at the cost of flexibility. There is a risk that people follow “rules” blindly without understanding the rationale.
- Education: Aims for a culture where experimenters understand the why behind best practices. This leads to better buy-in, allows teams to challenge methods, and highlights individual responsibility. It also prevents bottlenecking. If enforcement requires human review, it slows down development.
However, education requires a massive time investment. New joiners must be constantly onboarded, and without strict rules, there is no guarantee that bad practices won’t take over, especially when business incentives don’t align with good science.
Booking.com has always emphasized individual responsibility. A flexible system is better suited to our changing business needs. However, until a few years ago, this flexibility led to a lack of reporting on the quality of experiments. We had no systematic tracking, governance, or assessments of statistical validity.
The Evolution of Experiment Quality
To address quality issues, we launched several programs: Community Initiatives and Tool Enhancements.
1. Community Initiatives
To help teams understand the risks of sub-optimal experiments, we established three key programs:
- Experiment Bashes: Regular meetings where decision-makers review past experiments. They ask: What did we expect? (Hypothesis), What did we learn? (Business learnings), and How valid are the results?
- Experiment Reviews: Small groups review randomly chosen experiments, covering the hypothesis, setup, and final decision to provide actionable feedback to owners.
- Experiment Ambassadors: A network of experimentation enthusiasts, experts, and Data Scientists embedded within product teams. They advocate for high-quality experimentation within their specific tracks or departments.
2. Measuring Quality
Despite these initiatives, we were missing a way to formally define quality. Our Experiment Ambassadors took the lead to measure quality by focusing on three questions:
- Did the experiment include a power calculation?
- Did the experiment adhere to its planned runtime?
- Did the decision align with the hypothesis?
As this initiative grew, our Centralised Experimentation Team took notice. They developed a unified, three-pillar Experimentation Quality metric (Figure 1) focused on
- Design: e.g., power calculations
- Execution: e.g. runtime adherence
- Decision: e.g. alignment of results with hypothesis
The Experimentation Quality metric was integrated directly into our “ET” platform. (For a detailed description of the metric, please see this article).
To make monitoring experiment quality easy and transparent the Centralised Experimentation Team created a dashboard. Initial data revealed where the biggest problems with experimentation quality lie. For example, 80% of experiments did not have a power calculation.
In response, Experiment Ambassadors began engaging directly with Product Teams using these dashboards to investigate why standards weren’t being met. Additionally, our internal Experimentation Wiki provided deep dives into quality for continuous learning.
3. Improving Quality Through the Tool
Since Booking.com uses an in-house experimentation tool, we have the unique ability to update the platform itself to improve experimentation quality. Our goal was to make running high-quality experiments easy. We wanted to make it: obvious, simple, automatic, and the default.
Make it obvious — The Quality Tab: In the UI where we display the metric results of A/B test we introduced a “Quality tab” alongside every experiment (Figure 2), which highlights violations, explains the issue, and provides instructions on how to fix it, right when the user benefits the most from this information.

Make it easy: The Power Menu Insufficient statistical power was a major issue. Power is a complex concept, and teams struggled with the trade-off between velocity and detectability. We introduced a “Power Menu” (Figure 3) to simplify this, this power menu displays the runtimes alongside effects sizes, so users can make better-informed decisions on the tradeoffs between speed and detectability. The tool now sends specific alerts when an experiment is underpowered, linking directly to the menu so users can select an option with sufficient power.

Make it automatic: We also provided an option to avoid calculations entirely: Experimenters can opt in to an automatic power calculation. The tool performs an automatic power calculation after a data collection “burn-in” period and runs the test accordingly. While this reduces control over the exact runtime, it ensures sufficient power.
Make it the default: Furthermore, we found that providing strong defaults while creating an A/B test was one of our most successful interventions. Previously, users could explore data across many metrics, leading to “peeking” outside pre-registered data points. By defaulting to showing only the relevant decision data (and making other data points less convenient to access), we significantly improved quality.
The Results
The combination of a powerful tool and dedicated education has created a cycle of learning and consistent improvement. Since introducing these initiatives, the share of high-quality experiments has steadily risen (Figure 4)

Taking a deeper look, we have significantly mitigated issues within the ‘Good Design’ pillar (Figure 5). This represents the largest area of improvement, specifically regarding power calculations and pre-registration.
This means our experiments are now more appropriately powered. Higher quality setups allow us to detect smaller effects, giving us greater confidence that we are correctly measuring the impact of our interventions.

Conclusion
Standardising definitions, implementing reliable tracking and strong experiment defaults were critical steps for our program. Measuring Experimentation Quality (EQ) created the transparency and urgency needed to turn standards into a measurable metric. This shift allowed us to move beyond simple recommendations and establish formal Quality Targets that our product areas now own.
Additionally, by establishing a shared responsibility between Product teams, the Central Experimentation Team, and our Experiment Ambassadors, we have successfully scaled quality alongside volume. Transitioning Experimentation Quality from a recommendation to a business target has changed how we make decisions.
With high-quality experimentation, we ensure trustworthy results. We ship the better features to our customers and use reliable experiment learnings to steer our product strategy in the right direction.
Scaling Experimentation Quality at Booking.com was originally published in Booking.com ML & DS Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.





