7 Checks for Whether A/B Test Wins Hold Up After Launch

Published 9 November 2026 · Last updated and fact-checked November 2026

Short answer: Often they do not hold up at the size reported. In a 2026 KDD replication of eight A/B tests with a median of 2.4 million users each, Vermeer, Kohavi and colleagues found only two significant in the expected direction. No study I found gives a typical post-launch survival rate, so discount every win and keep a holdout.

Your CRO report lists six wins this quarter. Each one says plus 12% or plus 20%. Add them up and the form should be converting a third better than last year. Then you open the CRM and the number of sales calls looks the same as it did in spring.

That gap is not a mystery, and it is usually not fraud or a broken tag. It is what statistics does to small, noisy tests that get picked because they won. The research on it is solid. Most of it sits in conference papers that marketing teams never read.

On 9 November 2026 I read the five vendor pages that rank for this question, from Mida, Statsig, LogRocket, DEV Community and ZURB. They explain novelty effects and the winner's curse well. None of them reports the 2026 replication numbers, and none applies the evidence to a B2B site that gets 5,000 visitors a month to its demo page. This guide does both, as seven checks I run before I believe a win.

Key takeaways

• In the KDD 2026 replication by Lukas Vermeer, Ron Kohavi, Jakub Linowski and three co-authors, 2 of 8 tests were significant in the expected direction and 1 was significant in the opposite direction.

• Berman and Van den Bulte estimated that 18% to 25% of results significant at 5% on a commercial testing platform were null effects, across 4,964 effects from 2,766 experiments.

• A published claim that rounded button corners lifted clicks by 55% shrank by about two orders of magnitude in three tests with over 2,000 times more users, in a 2025 preprint by Kohavi, Linowski, Vermeer, Andrew Gelman, Guido Imbens and others.

• Google researchers Hohnhold, O'Brien and Tang showed the short-term effect measured in a test does not always predict the long-term effect after launch.

• Using the sample size rule in Kohavi and colleagues' 2009 guide, a page converting at 2% needs about 78,400 visitors per variant to detect a 10% relative lift, my calculation.

1. Was the test powered? Best for small-sample wins

Power is the chance a test detects a real effect of the size you planned for. Most experimenters aim for 80%. When power is far lower, a result can only reach significance by overstating the effect. Vermeer and colleagues believe many prior pattern estimates came from tests with power below 50%.

• Ask for: the minimum detectable effect set before launch

• Red flag: a test sized after it started

• Planned power: 80% is the common target

• Source of the rule: Kohavi, Longbotham, Sommerfield and Henne, 2009

Gelman and Carlin gave this problem two names. A Type S error is an estimate with the wrong sign. A Type M error is the factor by which a significant estimate overstates the true effect, which they call the exaggeration ratio. Both grow as power falls, and neither shows up in the p-value your testing tool prints.

Here is why that matters on a B2B site. Suppose the true lift from a new form layout is 3%. Your test has 4,000 visitors per arm. Most runs will show no significant result, and the few that do will show a lift big enough to clear the threshold, perhaps 15% or 20%. The tool reports one of those lucky runs as a win. You ship it and the real 3% arrives, if it arrives at all.

So the first thing I ask for is the minimum detectable effect, written down before the test began. If nobody set one, treat the reported lift as an upper bound, not an estimate.

Data point: Gelman and Carlin recommend estimating the probability of a wrong-direction estimate and the exaggeration ratio at the design stage.

"In noisy, small-sample settings, statistically significant results can often be misleading," Andrew Gelman and John Carlin wrote in Perspectives on Psychological Science.

2. Is the lift plausible? Best for wins borrowed from case studies

A plausibility check compares your reported lift with what large tests of the same kind of change have found. This matters most when a vendor case study or a past win is the reason you ran the test. In the KDD 2026 work, effects for four popular patterns came in substantially smaller than previously claimed.

• Patterns replicated: rounded buttons, page performance, coupon-code field, sticky call to action

• Median sample: 2.4 million users per experiment

• Significant as expected: 2 of 8

• Significant the other way: 1 of 8

The Trustworthy A/B Patterns project ran for over a year and a half as a community replication effort. Retailers and platforms, including METRO, Talabat and Coop Norge according to the author affiliations, each ran the same pattern on their own traffic. They chose tests with enough users to detect lifts between 0.3% and 2.2%.

The rounded corners case is the clearest. A 2023 Journal of Consumer Research study by Biswas, Abell and Chacko reported a 55% click-through lift from rounding square buttons, with a p-value of 0.037. Three replications with over 2,000 times more users found estimates about two orders of magnitude smaller, and none was significant. Two independent replications by Evidoo found similarly small effects, according to the preprint.

My rule from this: if a cosmetic change claims a double-digit lift, I assume most of it is noise until a second test agrees. Copy and offer changes can move more, but the same test applies.

Some intuitive claims "touted, including by A/B tool vendors and agencies" are "misleading, often badly so," Ron Kohavi, Alex Deng and Lukas Vermeer wrote in the KDD 2022 proceedings.

Data point: across all eight replications, Vermeer and colleagues found estimated effects substantially smaller than prior claims.

"Previously reported effects for these patterns are highly exaggerated," Vermeer, Kohavi, Linowski, Andreev, Dodin and Furuseth wrote in the KDD 2026 proceedings.

3. What share of your wins are false? Best for teams with a long win list

The false discovery rate is the share of significant results that are really zero effects. It is not the same as your 5% significance level. On one commercial platform, Berman and Van den Bulte estimated it at 18% to 25% for tests at 5% significance, and 28% to 37% at 10%.

• Data set: 4,964 effects from 2,766 experiments

• True nulls: about 70% of tested effects

• At 5% significance: 18% to 25% of wins are null

• At 10% significance: 28% to 37% of wins are null

The cause surprised me when I first read it. The authors found the high rate came mostly from the large share of ideas that do nothing, about 70%, and less from low power. Most changes we test have no effect. When most ideas are duds, even a well-run test lets a steady trickle of duds through as winners.

Look at the significance setting in your testing tool. Some tools and agencies report at 90% confidence, which is 10% significance. On Berman and Van den Bulte's numbers, that moves you from about one null win in five to about one in three.

Their practical advice was two-stage designs: test several variations, then retest the winner. For a B2B team, the cheaper version is to rerun any win you plan to build a quarter's roadmap on.

Data point: Berman and Van den Bulte used three different estimation methods and all three landed in the same range.

Berman and Van den Bulte wrote that "decision makers should expect one in five interventions achieving significance at 5% confidence to be ineffective when deployed in the field," in Management Science.

4. Was the winner picked from many? Best for multivariate and multi-arm tests

The winner's curse is the bias that comes from launching only the tests that won. The winning estimate includes its lucky noise, and that noise does not ship. Airbnb researchers Minyong Lee and Milan Shen showed the summed impact of launched features is biased upward and built a correction into Airbnb's reporting platform.

• Where it bites: tests with three or more variants, and quarterly roll-ups of wins

• Symptom: the sum of reported lifts beats the actual change in the top line

• Fix in the paper: a bias-corrected estimator

• Platform: Airbnb Experiment Reporting Framework

Picture five headline variants. The true lift of the best one is small. Each variant's measured result bounces around its true value. The one you crown is usually the one whose noise bounced highest, so its reported lift sits above where it will land after launch.

The same thing happens across a quarter. If a team runs 20 tests and reports the six winners, the roll-up adds six numbers that each picked up their own good luck. That is why the sum of a year's wins so rarely shows up in revenue.

I handle this two ways. First, I report a quarter's wins as a range, with the low end at roughly half the claimed lift until a holdout confirms it. That halving is my working rule, not a figure from any study. Second, for the one or two wins that matter most, I keep a small group on the old version after launch.

Data point: Lee and Shen describe the bias as a statistical selection effect created by choosing significant experiments to launch.

"Usually the ones with significant successful results are chosen to be launched into the product," Minyong Lee and Milan Shen wrote in the KDD 2018 proceedings.

5. Did anyone stop the test early? Best for tests watched on a dashboard

Peeking means checking a running test and stopping it the first time it looks significant. Every look is another chance for noise to cross the line. Johari, Koomen, Pekelis and Walsh built always valid p-values for Optimizely because users of A/B testing software are known to monitor results continuously.

• Risk: a fixed-horizon test checked daily

• Safer setup: a planned end date, or sequential statistics built for peeking

• Ask your vendor: which method its significance number uses

• Paper: KDD 2017, deployed at Optimizely

Nobody sets out to peek. A stakeholder opens the dashboard on day four, sees green, and asks why the winner is not live yet. The test ends there. If the tool used a classical fixed-horizon test, that green light had a much higher false positive rate than the 5% printed on the screen.

Before trusting a win, I check two dates: the planned end date and the actual one. If they differ, I ask whether the tool uses sequential testing. Most current platforms offer it, but it is not always the default, and older tests in a win log may predate it.

The other fix is cultural. Agree before launch that nobody stops a test on a good-looking day. Write the end date into the test brief and treat an early stop as a new hypothesis, not a result.

Data point: Johari and colleagues report simulations and numerical studies on Optimizely data showing better detection than traditional methods.

Johari and colleagues wrote that "traditional p-values and confidence intervals give unreliable inference" when users monitor tests continuously, in the KDD 2017 proceedings.

6. Does the lift last? Best for visible design changes

A novelty effect is a lift that comes from a change being new rather than better. Returning visitors click the unfamiliar button because it is unfamiliar. Google researchers Hohnhold, O'Brien and Tang showed the short-term effect measured during a test does not always predict the long-term effect once users adjust.

• Most exposed: sites with many returning visitors, such as customer portals

• Less exposed: paid traffic landing pages, where most visitors are new

• What to watch: lift by week, split by new and returning visitors

• Paper: Google, KDD 2015

For most B2B marketing sites, novelty is a smaller risk than the first four checks. The bulk of traffic to a demo page or a pricing page is new, and new visitors have no old version to compare against. Novelty matters more on pages your existing customers see often, like a login page, a resource hub or an in-app upgrade prompt.

The check is cheap. Plot the lift week by week rather than as one number. A real effect stays roughly level. A novelty effect starts high and slides. If your tool can split by new and returning visitors, do that too: a lift that exists only for returning visitors is a warning sign.

Hohnhold and colleagues went further and modelled how users learn over time, so they could predict the long-term effect from a shorter test. Few marketing teams need that. They do need the habit of looking at the shape, not only the total.

Data point: Hohnhold, O'Brien and Tang describe the long-term effect as the final impact after a product has fully launched and users have changed their behavior.

"The short-term effect is not always predictive of the long-term effect," Henning Hohnhold, Deirdre O'Brien and Diane Tang wrote in the KDD 2015 proceedings.

7. Did the metric that pays you move? Best for proving pipeline

A surrogate metric is something easy to move that you hope predicts something valuable, like clicks predicting leads. Even with a median of 2.4 million users, the KDD 2026 team lacked power for revenue per user and purchase conversion and had to use surrogates such as click-through and add-to-cart rate.

• Usually tested: clicks, form starts, form submits

• Usually wanted: qualified leads, pipeline, revenue

• Gap: each step down the ladder needs more traffic

• After launch: compare lead quality, not only lead count

This is the check that explains most of the missing pipeline on B2B sites. A test that moves form submissions by 15% can do it by pulling in more low-intent people. Sales sees more names and no more deals. The test was not wrong about submissions. It answered a smaller question than the one you cared about.

If a retailer with 2.4 million users per test cannot power purchase conversion, a B2B site with 5,000 monthly visitors cannot power sales-accepted leads. So I treat tests as evidence about behaviour on the page, then check the business metric after launch with a before and after comparison and, where traffic allows, a holdout.

A holdout is a small group, say 5% to 10% of visitors, that keeps seeing the old version after you ship the winner. It is the only clean way to see whether the effect survived. If your traffic is too low for that, compare lead quality fields in your CRM for the eight weeks before and after launch, and write down what else changed in that window.

Data point: Vermeer and colleagues ran each test at 80% power for pre-selected minimum detectable effects of 0.3% to 2.2%.

"Even at this scale, we did not have enough power for key business metrics," Vermeer and his co-authors wrote in the KDD 2026 proceedings.

How much traffic does a B2B test need?

A B2B page needs far more visitors to test small lifts than most teams expect. Using the rule of thumb in Kohavi and colleagues' 2009 guide, 16 times the variance divided by the squared effect per variant, a page converting at 3% needs about 51,733 visitors per variant to detect a 10% relative lift.

These are my calculations, at 80% power and 5% significance, two-sided, from the Kohavi, Longbotham, Sommerfield and Henne formula.

At 5,000 visitors a month to a demo page converting at 2%, detecting a 10% lift takes about 31 months. Detecting a 30% lift takes under four. That is why low-traffic B2B tests mostly report big wins: only big apparent effects can reach significance, and on Gelman and Carlin's argument those are the ones most likely to be exaggerated.

There is an argument for small tests. Azevedo, Deng, Montiel Olea, Rao and Weyl measured effect sizes on Bing's experimentation platform and found extremely fat tails, meaning rare ideas produce very large gains. When that holds, they show that trying more but smaller interventions beats a few high-powered tests. The catch is that small tests then need replication before you bank the result.

"A lean experimentation strategy of trying more but smaller interventions is optimal" when the distribution of effects is very fat tailed, Azevedo and his co-authors wrote in the Journal of Political Economy.

What the research measures and what nobody has measured

This table separates measured findings from things that are only commonly claimed. Every value comes from the paper named in its row. The one figure buyers ask for most, the typical share of an A/B test lift that survives launch, has no peer-reviewed estimate that I could find.

You will see figures online such as "most A/B test wins are illusory" or a fixed percentage of lift that decays. I could not trace a typical survival percentage to a peer-reviewed source, so I do not give one. The honest answer is that survival depends on power, selection, peeking and the metric, which is why the seven checks exist.

Also note the scale. The KDD 2026 tests ran on retail and delivery sites with millions of users. A B2B site runs tests at a tiny fraction of that, which makes every check in this guide more important, not less.

Methodology

I ran this research on 9 November 2026. I searched OpenAlex, Crossref, Semantic Scholar and Google for peer-reviewed work on A/B test replication, false discovery, selection bias, peeking and long-term effects, and read the abstracts and author pages quoted here. The limits: I read abstracts, not every full text, and the rounded corners figures come from a preprint.

Every number in the article maps to an entry in a fact sheet kept with the draft. Quotes are verbatim from abstracts or the author's paper page. The sample size table is my arithmetic from a published formula, not a measured result. I read five competing pages on 9 November 2026 to check what they cover: Mida, Statsig, LogRocket, DEV Community and ZURB. Two others did not load.

Which check matters most for your test?

The right first check depends on how the test was run. Use this matrix to pick where to start. A cosmetic change on a page with under 10,000 monthly visitors needs checks 1 and 2 before anything else. A multi-variant headline test needs check 4. A win reported in clicks needs check 7.

The sample size calculator that goes with this guide takes your baseline rate, the lift you want to detect and your monthly traffic, then returns the months a test would need.

Frequently asked questions

Why did my A/B test winner not increase conversions after launch?

The most common reasons are low power, picking the best of several variants, stopping early and measuring a click instead of a lead. In the KDD 2026 replication, only 2 of 8 tests of popular patterns were significant in the expected direction. Discount the reported lift and check pipeline.

What is the winner's curse in A/B testing?

It is the upward bias created when you launch only the tests that won. The winning estimate includes lucky noise that does not ship. Airbnb researchers Lee and Shen documented this selection bias and proposed a correction. Expect the shipped effect to be smaller than the reported one.

How many A/B test wins are false positives?

On one commercial platform, Berman and Van den Bulte estimated that 18% to 25% of wins at 5% significance were null effects, and 28% to 37% at 10%. Their sample covered 4,964 effects from 2,766 experiments. Your rate depends on how many of your ideas truly work.

How long should I run an A/B test on a low-traffic B2B site?

Run it until it reaches the sample you calculated before launch, never until it looks good. At 5,000 monthly visitors and a 2% baseline, a 30% lift needs about 3.5 months by the Kohavi 2009 rule. Smaller lifts can take years.

Should I keep a holdout group after launching a winner?

Yes, for any change you plan to build on. A holdout of 5% to 10% of traffic keeps seeing the old version, so you can measure what survived. Google researchers Hohnhold and colleagues showed short-term test effects do not always predict long-term ones.

How I run tests for B2B clients

I run conversion work for US B2B teams inside fixed-price website projects at Demand Designer. On a typical B2B site, the traffic math rules out most small tests, so I start with research and the leaks it finds. I test only changes big enough to measure, and I check pipeline after launch rather than trusting the test report alone.

The Pipeline Leak Scorecard is the free version of that first pass. It scores the parts of your site where leads get lost, so you can decide what is worth testing and what is just worth fixing. If your tests report wins but your CRM does not move, check your setup against my guide to website lead tracking and the B2B conversion rate benchmarks before you run another one.

Sources

Peer-reviewed research

• Azevedo, E. M., Deng, A., Montiel Olea, J. L., Rao, J. M., and Weyl, E. G. (2020). A/B testing with fat tails. Journal of Political Economy, 128(12).

• Berman, R., and Van den Bulte, C. (2022). False discovery in A/B testing. Management Science, 68(9).

• Gelman, A., and Carlin, J. (2014). Beyond power calculations: Assessing Type S (sign) and Type M (magnitude) errors. Perspectives on Psychological Science, 9(6).

• Hohnhold, H., O'Brien, D., and Tang, D. (2015). Focusing on the long-term: It's good for users and business. Proceedings of the 21st ACM SIGKDD Conference on Knowledge Discovery and Data Mining.

• Johari, R., Koomen, P., Pekelis, L., and Walsh, D. (2017). Peeking at A/B tests: Why it matters, and what to do about it. Proceedings of the 23rd ACM SIGKDD Conference on Knowledge Discovery and Data Mining.

• Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., and Xu, Y. (2012). Trustworthy online controlled experiments: Five puzzling outcomes explained. Proceedings of the 18th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.

• Kohavi, R., Deng, A., and Vermeer, L. (2022). A/B testing intuition busters. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.

• Kohavi, R., Longbotham, R., Sommerfield, D., and Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1).

• Lee, M. R., and Shen, M. (2018). Winner's curse: Bias estimation for total effects of features in online controlled experiments. Proceedings of the 24th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.

• Vermeer, L., Kohavi, R., Linowski, J., Andreev, A., Dodin, M., and Furuseth, J. (2026). Trustworthy A/B patterns and the winner's curse: Lessons from eight large-scale replications. Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, V.2, 7533 to 7544.

Preprints and other sources (not peer reviewed), read 9 November 2026

• Kohavi, R., Linowski, J., Vermeer, L., Boisseranc, F., Furuseth, J., Gelman, A., Imbens, G., and Rajagopal, R. (2025). Power analysis is essential: High-powered tests suggest minimal to no effect of rounded shapes on click-through rates. arXiv preprint 2512.24521.

• Vermeer, L. Trustworthy A/B patterns and the winner's curse, publication page with abstract.