At Booking.com, thousands of experiments run simultaneously across highly heterogeneous users, from individual travellers to large travel agencies. This means our experiment data regularly contains legitimate but extreme values. When these go unhandled, they distort the statistical conclusions we draw, leading us to scale ideas that don’t create value, or to discard ones that do. So, we need outlier handling methods that are reliable, automated, and applicable across diverse metrics without manual intervention.
The Problem
When extreme values are present in experiment data, they can compromise the estimation of average treatment effects (ATE), leading to unreliable test results and reduced statistical power. Even a single observation can inflate variance enough to mask a real effect or produce a spurious one. In practice, this means we risk shipping changes that appear positive but are not, or killing promising features because noise masked their real effect. At Booking.com’s scale, this increase in false conclusions quickly compounds into a meaningful impact on customer experience and business outcomes.
A Diagnostic Tool: the Permutation Test
One way to assess whether extreme values are distorting results is the permutation test. By permuting over experiment data, we generate hundreds of simulated AA experiments where we know the ground truth: there is no real effect. Plotting the resulting p-values, we expect a uniform distribution. If it instead looks skewed, the underlying data distribution is compromising the validity of results.

Simulation Evidence: What Drives Failure?
We ran AA permutation tests across a range of simulated data distributions to understand when they fail (i.e. not show a uniform p-value distribution). For distributions without extreme values, the test passes reliably. As outlier magnitude and frequency increases, the test begins to fail. First intermittently, where other distribution characteristics like zero inflation play a role, and then consistently once the most extreme values are sufficiently far from the rest of the distribution.
A Simple Summary Statistic: the Max-to-99.9th Percentile Ratio
To summarise this relationship, we measured how many times larger the most extreme observation is relative to the bulk of the distribution. We found the ratio of the maximum value to the 99.9th percentile worked well for our data distribution, though the general principle held with other ratios: if the maximum is orders of magnitude beyond most of the distribution, the permutation test will likely fail.

A Solution: Winsorization
Winsorization, capping extreme values at a threshold rather than removing them, preserves each data point’s presence while limiting its influence on the mean. Across our simulated and real distributions, metrics that previously failed the permutation test pass after enough winsorization. Combined with its ease of implementation and scalability, this made it a natural fit for our scale, as reliable outlier handling directly improves the quality of decisions we make for our customers.
Threshold Types
Two winsorization approaches exist:
- Relative thresholds: capping at a percentile of the observed distribution (e.g. 99th percentile). These adapt to each metric’s distribution, which is useful when distributions vary substantially across experiments.
- Absolute thresholds: capping at a fixed value informed by business logic (e.g. 1000 transactions per user). These are easier to interpret and communicate to stakeholders.
Which strategy works best depends on the context. In practice, both deliver clear benefits: improved statistical validity (higher AA test pass rates), improved power (detecting real effects that would otherwise be lost in noise), and scalability (no manual intervention required per experiment). Together, these translate into more trustworthy experiment outcomes and better-informed product decisions.
Sensitivity: Choosing a Threshold
Many different thresholds restore statistical validity, from moderate to aggressive thresholds. The challenge is choosing among them. The core tradeoff is between variance reduction (more aggressive capping reduces noise) and bias introduction (capping alters real data).
To avoid choosing a threshold that happens to work for one experiment but not others, we evaluated thresholds across multiple experiments simultaneously. For each candidate threshold, we ran the permutation test on a set of experiments and retained only thresholds for which the test consistently passes as this is our primary constraint. Among these, we then computed the mean and standard deviation of the capped metric across the experiments. This produces a distribution per threshold rather than a single estimate so we can get a heuristic for removing noise without distorting the signal too much. A good threshold sits where the standard deviation has decreased meaningfully from its uncapped value, but the mean has not yet shifted substantially.

Limitations
Winsorization is effective but not without risks:
- When the intervention genuinely moves the tail: If a treatment creates legitimate extreme values (e.g. a feature that drives a few users to book much more), winsorization will mask this real effect and so the winsorized metric might not be suitable. However, is is a product concern: experimenters who believe their intervention will target this tail should pre-register this hypothesis and validate it by defining eligibility criteria for this segment (e.g. high frequency bookers).
- Threshold sensitivity to experiment duration: An absolute threshold appropriate for a 2-week experiment may be overly restrictive for a 2-month one, since the metric distribution changes over time. Thresholds should be tested across a range of experiment runtimes and types.
- The bias-variance tradeoff is hard to measure: The analysis described above provides a practical heuristic, but the comparisons it relies on are limited. The non-winsorized mean is not a reliable ground truth if outliers are biasing it, and standard deviation will decrease by design with capping. Stability of the estimate across experiments might be a useful indication, but ultimately the “right” threshold remains a judgement call informed by domain knowledge and the specific metric’s context.
Open Questions
- Updating thresholds: For example in sequential testing, as experiments are monitored over time, should the winsorization threshold be adapted as more data arrives?
- Using Different Methods: Huber-loss based estimators are more principled handling of outliers but with increased complexity and reduced scalability.
Outlier handling is not a solved problem, but a combination of diagnostics and a scalable approach meaningfully improves the quality of decisions that AB testing enables. At Booking.com’s scale, where thousands of experiments inform how we serve hundreds of millions of travellers, this reliability translates directly into better outcomes for our customers.
Acknowledgements
Would like to thank Raffaele Rainone for the valuable reviews and initial input on the direction of this post!
Outlier Handling at Scale in Experimentation was originally published in Booking.com ML & DS Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.





