7 Tests: Can AI Simulated Users Replace A/B Testing?

Published 10 November 2026 · Last updated and fact-checked November 2026

Short answer: Not yet. The best published result, Shopify's SimGym preprint, matched the direction of real add-to-cart shifts in 77% of 50 shops, but the size of the effect correlated only at 0.55. Every study I found predicts which variant wins more reliably than by how much. Use simulated users to screen ideas, then confirm with real visitors.

Shopify now lets merchants try theme changes on AI shoppers before real customers see them. A merchant in one of the busiest threads on this asked the obvious question after the simulation disagreed with his own test result. Would you ship one based on that?

I read the research to answer it. On 10 November 2026 I went through the five pages that rank for this question, from Neuroflash, Experimentation Nation, PM Toolkit, Spotify Engineering and Learnopoly. Each covers one study or one vendor. None puts the published accuracy figures side by side, says plainly that they predict direction rather than effect size, or works out what that means for a B2B site with 3,000 visitors a month to its demo page. This guide does all three, one published test at a time.

One warning up front. Five of the seven studies here are preprints or company research posts. The peer-reviewed work in this guide is about whether language models can stand in for people in general, not about website tests specifically. No peer-reviewed field study has yet validated simulated users against real B2B lead form tests.

Key takeaways

• SimGym, from 20 Shopify researchers, matched the sign of real add-to-cart shifts in 77% of 50 shops, with a 0.55 correlation on effect size.

• SimAB picked the right winner in 67% of 47 historical A/B tests, rising to 83% when its agents agreed quickly.

• Amazon researchers Hut and Masoero found a sign overlap of 0.70 on 67 marketing tests, with effect sizes that "systematically overshoot".

• Spotify Engineering found raw model predictions recovered only 39% of the real treatment effect across thousands of Upworthy headline tests.

• In Nature Machine Intelligence, Angelina Wang and colleagues showed across 3,200 people and 16 identities that language models misportray and flatten demographic groups.

1. SimGym from Shopify: best for visual theme changes on stores

SimGym runs AI shoppers through two versions of a live storefront and counts add-to-cart. It suits ecommerce teams testing visual theme changes. On 50 Shopify shops it matched the direction of the real shift 77% of the time, but its effect size estimates correlated with reality at only 0.55.

• Built by: 20 researchers at Shopify

• Sample: 50 shops, each with a control and a treatment storefront

• Metric: add-to-cart rate, not revenue or leads

• Direction match: 77%

• Effect size correlation: 0.55

• Status: arXiv preprint, May 2026

SimGym is the most serious attempt so far, and it is the one merchants can now reach through Shopify's own tools. It builds buyer personas from each shop's real clickstream, so the agents shop the way that store's visitors shop. Then a vision-language model drives a real browser through the control and treatment stores.

The headline result is good for direction. In 77 of every 100 shops, the agents agreed with real buyers about whether add-to-cart went up or down. On size, a correlation of 0.55 means the agents roughly track which changes matter more, with a lot of scatter. The paper also shows how fragile this is: with the agents' memory switched off, direction fell to 42%, worse than a coin flip, and correlation collapsed to about zero.

Two limits matter for a B2B team. The tests were visual theme changes on stores, and the metric was add-to-cart. Neither is a demo request from a buying committee.

Data point: the SimGym framework preprint by Han Li and 19 co-authors at Shopify reports 77% directional alignment and a 0.55 Pearson correlation across the 50-shop set.

2. SimAB: best for screening designs on low-traffic pages

SimAB shows design screenshots to persona-based AI agents and asks which version they prefer. It suits teams that cannot get enough traffic for a live test. Against 47 historical A/B tests with known results it picked the right winner 67% of the time, and 83% on the cases where agents agreed fast.

• Built by: Tim Rieder, Mustafa Doga Dogan and seven co-authors

• Input: screenshots plus a conversion goal

• Sample: 47 past tests of layout and content

• Accuracy: 67% overall, 83% high-confidence

• High-confidence share: 60% of tests

• Status: arXiv preprint, March 2026

SimAB is the study that speaks most directly to B2B sites, because its authors built it for the cases where traffic is the problem. In interviews with experimentation practitioners they listed low-traffic pages, many-variant comparisons and small tweaks as places where real tests stall.

The method is simple. Agents look at screenshots, state a preference and give a reason. A test counts as high confidence when a significant preference appears within 70 agents. About 60% of the tests reached that bar.

Here is the catch for anyone deciding what to ship. SimAB tells you which version agents prefer, not how much more often real people would convert. A 67% hit rate also means about one wrong call in three. One filtered analysis in the same paper reports 75% accuracy rather than 83%, so I would treat the higher figure as the best case.

Data point: Rieder and colleagues define high-confidence tests as those reaching a significant result within at most 70 agents, and report that 60% of all tests met it.

3. Amazon's simulated trial: best for teams with past test data

Amazon's S-RCT framework asks AI agents with behavioural profiles to react to a described change, then compares the simulated effect with the real one. It suits large teams with a history of past tests. On 67 marketing tests it got the direction right 70% of the time but overstated effect sizes.

• Authors: Stefan Hut and Lorenzo Masoero, Amazon

• Sample: 67 historical marketing A/B tests

• Direction match: sign overlap 0.70

• Effect size: systematically overshoots

• After calibration: about 77 times less squared error

• Status: conference paper, COLM 2026

This paper is the most useful one for understanding where the errors come from. Hut and Masoero split the gap into two parts: the agent being a poor copy of a real customer, and the noise from simulating only a sample. That split tells you what to fix.

The baseline result is a familiar shape by now. With an off-the-shelf model, the agents got the direction of the effect right about seven times in ten. The sizes were too big. A simulator that overstates effects is dangerous in a specific way: every idea looks like a winner worth building.

The fix was calibration. Using data from before each test, they cut the squared prediction error by about 77 times. A design where each agent sees both versions also cut standard errors by about 2.4 times. Both fixes need the real past data that a small B2B team usually does not have.

Data point: Hut and Masoero report a baseline sign overlap of 0.70 on 67 historical marketing A/B tests and an approximately 77 times reduction in squared prediction error after two-phase calibration.

4. Spotify's surrogacy test: best for text and headline variants

Spotify's data scientists asked when a model's prediction can legally stand in for a person's response in a test. It suits teams testing short text variants such as headlines. Raw predictions on Upworthy headline tests recovered only 39% of the true effect, so treatments looked less than half as effective as they were.

• Authors: Sebastian Ankargren, Joel Persson and Mårten Schultzberg

• Data: Upworthy Research Archive, thousands of headline tests

• Raw result: 39% of the real effect recovered

• Linear calibration: off by 3.8 standard errors

• Tree-based calibration: within sampling error

• Status: company research post, August 2026

This is the most important study in the list, because it explains why the others keep finding direction without size. A real randomized test proves cause by design. A model's prediction only proves it by assumption. The Spotify team spells out the two assumptions. First, the prediction must capture everything about the change that matters to people. Second, the link between prediction and behaviour must stay the same for the new change.

Raw predictions failed both ways. The bias was not random noise. It pulled every effect toward zero. Calibrating the predictions against real test results fixed this, but only with flexible machine learning methods, and only on data that looked like the past.

Their conclusion is the one I would put on a wall. The assumptions can never be proven for a change you have not tested before. A new layout, a new pricing page or a new form is exactly that kind of change.

Data point: the Spotify Engineering team reports that raw gpt-4o-mini predictions recovered 39% of the observed human treatment effect in the Upworthy archive.

5. AgentA/B: best for walking agents through a whole flow

AgentA/B sends AI agents with personas through a live website to search, click, filter and buy. It suits teams that want to watch how a multi-step flow behaves. In a 1,000-agent study on Amazon.com, agent behaviour looked human-like, and the authors frame it as a complement to real tests.

• Authors: Yuxuan Lu, Dakuo Wang and ten co-authors

• Sample: 1,000 agents, between-subject design

• Site: Amazon.com

• Finding: human-like behaviour patterns

• Framing: a complement, not a replacement

• Status: arXiv preprint, 2025

AgentA/B is the study most people mean when they picture AI shoppers. The agents do not just look at a screenshot. They navigate a real page and act on it, step by step, the way the generative agents in Park and colleagues' 2023 UIST paper planned their days in a simulated town.

The authors started where I would start, with interviews. Six experienced practitioners described the same bottlenecks: tests need lots of live traffic and take a long time to read. The tool answers that with speed.

What it does not give you is a tested accuracy figure on B2B pages. The 1,000-agent study compared agent behaviour with real shopping behaviour on one large retail site and found the patterns were similar. That is useful for spotting a broken step in a checkout or a confusing filter. It is not evidence about whether a new demo form converts more people.

Data point: Lu and colleagues ran a between-subject simulated A/B test with 1,000 LLM agents on Amazon.com and compared agent behaviour with real human shopping behaviour.

"Believable proxies of human behavior can empower interactive applications ranging from immersive environments to rehearsal spaces for interpersonal communication to prototyping tools," Joon Sung Park and his Stanford and Google co-authors wrote in the UIST 2023 proceedings.

6. Persona fidelity research: best for knowing who the agents are

Every simulated test depends on personas: the agent is told to act as a certain kind of buyer. It suits anyone choosing an audience setting in a simulation tool. Peer-reviewed work shows models can copy group-level survey answers, yet also misportray and flatten groups, tested on 3,200 people across 16 identities.

• Optimistic case: Argyle and colleagues, Political Analysis, 2023

• Cautious case: Wang, Morgenstern and Dickerson, Nature Machine Intelligence, 2025

• Review: Dillion and colleagues, Trends in Cognitive Sciences, 2023

• Wang sample: 3,200 participants, 16 identities, 4 models

• Takeaway: fidelity holds for averages, fails for distinct groups

This is where the peer-reviewed evidence sits, and it points both ways. Lisa Argyle's Brigham Young University team showed that GPT-3, given thousands of real survey backstories, produced answers whose patterns matched human subgroups closely. They called this algorithmic fidelity.

Angelina Wang, then at Stanford, and her co-authors tested the other side. Across four models and human studies with 3,200 people, the models both misportrayed groups and flattened the differences inside them. Their techniques reduced the harm but did not remove it.

For B2B, think about who signs the contract. A buying committee might include a finance lead, an IT lead and an operator, each reading the same page for different reasons. A persona that sounds like the average of all three tells you very little about any one of them.

Data point: Wang, Morgenstern and Dickerson tested four LLMs through human studies with 3,200 participants across 16 demographic identities.

"We urge caution in use cases in which LLMs are intended to replace human participants whose identities are relevant to the task at hand," Wang, Morgenstern and Dickerson wrote in Nature Machine Intelligence.

"We explore whether and when language models might replace human participants in psychological science," Danica Dillion and her University of North Carolina and Allen Institute co-authors wrote in Trends in Cognitive Sciences.

"Proper conditioning will cause it to accurately emulate response distributions from a wide variety of human subgroups," Argyle and her co-authors wrote of GPT-3 in Political Analysis.

7. Variance reduction in live tests: best for low-traffic B2B sites

If the reason to simulate is too little traffic, there is an older, proven fix: make the real test more sensitive. It suits B2B sites that cannot reach a sample in a reasonable time. Microsoft's CUPED method used pre-test data to cut variance by about 50% on Bing, like doubling the traffic.

• Authors: Alex Deng, Ya Xu, Ron Kohavi and Toby Walker, Microsoft

• Venue: WSDM 2013, peer reviewed

• Result: about 50% less variance on Bing metrics

• Effect: same power with half the users or half the time

• Needs: data on each visitor from before the test

This is the item that most simulation pitches leave out. The problem they solve is real: a small B2B page cannot gather enough visitors. But the answer does not have to be fake visitors. It can be a better use of the real ones.

Here is my arithmetic, using the sample rule in Kohavi and colleagues' 2009 guide. A demo page converting at 3% needs about 12,933 visitors per variant to detect a 20% relative lift. At 3,000 visitors a month split two ways, that is about 8.6 months. Halve the variance and it is about 4.3 months.

The Spotify team makes the same point from the other direction. Model predictions can help real tests by filtering out weak ideas before they take a test slot, or by serving as a variable that reduces variance. That is the honest role for simulated users today: a screen in front of a real test, not a replacement for it.

"Controlled experiments embody the best scientific design for establishing a causal relationship between changes and their influence on user-observable behavior," Ron Kohavi and his Microsoft co-authors wrote in Data Mining and Knowledge Discovery.

Data point: Deng, Xu, Kohavi and Walker report CUPED reduced variance by about 50% on Bing's experimentation system.

"We can reduce variance by about 50%, effectively achieving the same statistical power with only half of the users, or half the duration," Deng and his Microsoft co-authors wrote in the WSDM 2013 proceedings.

Do simulated users predict the winner or the size of the win?

Simulated users predict which version wins far better than how much it wins by. Across the four studies with comparable figures, direction matched real tests 67% to 77% of the time, while effect sizes were too small in one study and too large in another. Never treat a simulated lift as a forecast.

Notice the second to last row. Real tests are not perfect either. Berman and Van den Bulte estimated that 18% to 25% of significant wins on one commercial platform were null effects. So a fair comparison is not simulation against truth. It is simulation against a real test that is also sometimes wrong, but wrong in ways statistics can measure.

"Decision makers should expect one in five interventions achieving significance at 5% confidence to be ineffective when deployed in the field," Ron Berman and Christophe Van den Bulte wrote in Management Science.

You will see vendor pages claiming 85% to 98% accuracy for synthetic audiences. I could not trace those figures to a published validation with a stated sample, so I do not repeat them as fact.

Methodology

I ran this research on 10 November 2026. I read the abstracts and results sections of the four preprints and the Amazon paper, the full Spotify post, and the abstracts of seven peer-reviewed papers, each checked on Crossref. The limits: most website-specific evidence is preprints, and none of it covers B2B lead forms.

Every number in this guide comes from the source named next to it. The months-to-detect figures are my arithmetic from the published Kohavi formula at 80% power and 5% significance, two-sided, not a measured result. I read five competing pages on 10 November 2026 to check what they cover. The Neuroflash page did not render its text for my word counter.

Which tool fits your situation?

The right use of a simulated test depends on your traffic and what you are changing. If your page gets under 5,000 visitors a month, use simulation to cut a long list to two versions, then run one real test with variance reduction. Over 50,000, simulation mostly saves test slots.

The calculator that goes with this guide takes your monthly visitors, baseline rate and target lift. It shows the months a real test needs, with and without variance reduction. That tells you whether simulation solves a real problem for you.

Frequently asked questions

Can AI simulated users replace A/B testing?

No. The published studies predict which variant wins in about 67% to 77% of cases and get effect sizes wrong. Spotify's data scientists showed the assumptions needed can never be proven for a new change. Use simulation to screen ideas, then confirm the finalist with real visitors.

How accurate is Shopify SimGym?

In Shopify's own preprint, SimGym matched the direction of real add-to-cart shifts in 77% of 50 shops, with a 0.55 correlation on effect size. Without agent memory, direction dropped to 42%. The study covered visual theme changes on stores, so it says little about B2B lead forms or pricing pages.

Are synthetic users good for user research?

They can help draft questions and spot obvious usability problems, but they do not represent specific groups well. In Nature Machine Intelligence, Wang and colleagues found models misportrayed and flattened 16 identities across 3,200 participants. Talk to real buyers before you trust a persona's answer.

What should I do instead if my site has too little traffic to test?

Test fewer, bigger changes and make each real test more sensitive. Microsoft's CUPED method cut variance by about 50% on Bing, like doubling traffic. Use simulated users to narrow the options first, and pick changes large enough to measure within six months.

How I use simulated tests for B2B clients

I run conversion work for US B2B teams inside fixed-price website projects at Demand Designer. On most B2B sites the traffic math rules out small tests, so I start with research into where leads get lost, fix what is plainly broken, and test only changes big enough to measure with real visitors.

Simulated users have a place in that process: I use them to throw out weak drafts early. I do not use them to decide what ships.

"Use simulated users to screen ideas, then confirm with real visitors," wrote Joshua Jackai, founder of Demand Designer.

The Pipeline Leak Scorecard is the free version of my first pass. If you already test, read my guide on whether A/B test wins hold up after launch and the B2B conversion rate benchmarks before you trust any result, simulated or real.

Sources

Peer-reviewed research

• Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., and Wingate, D. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3).

• Berman, R., and Van den Bulte, C. (2022). False discovery in A/B testing. Management Science, 68(9).

• Deng, A., Xu, Y., Kohavi, R., and Walker, T. (2013). Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. Proceedings of the Sixth ACM International Conference on Web Search and Data Mining.

• Dillion, D., Tandon, N., Gu, Y., and Gray, K. (2023). Can AI language models replace human participants? Trends in Cognitive Sciences, 27(7).

• Kohavi, R., Longbotham, R., Sommerfield, D., and Henne, R. M. (2009). Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18(1).

• Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. (2023). Generative agents: Interactive simulacra of human behavior. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology.

• Wang, A., Morgenstern, J., and Dickerson, J. P. (2025). Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence, 7(3).

Preprints, conference papers and company research (not peer-reviewed journals), read 10 November 2026

• Ankargren, S., Persson, J., and Schultzberg, M. When can LLMs replace humans in A/B tests? Spotify Engineering, 13 August 2026.

• Castelo, A., Zanjani Foumani, Z., Fan, A., and 17 co-authors. (2026). SimGym: Traffic-grounded browser agents for offline A/B testing in e-commerce. arXiv preprint 2602.01443.

• Hut, S., and Masoero, L. (2026). Can AI agents simulate A/B test outcomes? A validation framework for agentic experimentation. Conference on Language Modeling (COLM) 2026.

• Li, H., Malik, V., Zanjani Foumani, Z., and 17 co-authors. (2026). SimGym: A framework for A/B test simulation in e-commerce with traffic-grounded VLM agents. arXiv preprint 2605.19219.

• Lu, Y., Hsu, T.-Y., Gu, H., and nine co-authors. (2025). AgentA/B: Automated and scalable web A/B testing with interactive LLM agents. arXiv preprint 2504.09723.

• Rieder, T., Schneider, M., Truss, M., and six co-authors. (2026). SimAB: Simulating A/B tests with persona-conditioned AI agents for rapid design evaluation. arXiv preprint 2603.01024.