Welcome.AI
    Skip to content
    Machine Learning

    Squoosh Bench: Insights from 318 Historical A/B Tests

    This research focuses on how experimentation teams can make informed decisions about website changes by utilizing forecasts generated from simulated user behavior. The study analyzed 412 historical A/...

    arxiv.org•November 20, 2026•3 min read

    Key Facts

    • Leverage accurate forecasting to prioritize website changes that drive user engagement.
    • Implement Squoosh UX-1 for reliable predictions when selecting A/B tests to minimize risk.
    • Utilize historical A/B test data to refine experimentation strategies and improve decision-making.
    • Adopt a data-driven approach to enhance the effectiveness of website modifications.
    • Encourage collaboration between experimentation teams and data analysts to optimize forecasting accuracy.

    Summary

    Paper: Squoosh Bench: Evaluating UX-1 on 318 Historical Website A/B Tests

    Authors: Rohith Pasula, Johari Dramiga, Viraj Shah, Suzan Halim, Tyler Dooskin

    Executive Summary

    This research focuses on how experimentation teams can make informed decisions about website changes by utilizing forecasts generated from simulated user behavior. The study analyzed 412 historical A/B tests conducted on websites, narrowing down to 318 tests that lasted longer than 21 days. The goal was to see if predictions made by different systems about the potential success of website changes could aid in selecting which changes to test in a live environment.

    Four forecasting systems were evaluated based on their ability to predict the outcomes of these tests. The analysis involved masking page roles and withholding recorded outcomes during the prediction phase to ensure unbiased results. Among the systems tested, Squoosh UX-1, which is specifically fine-tuned using human browsing data, demonstrated a high level of accuracy. It correctly classified 267 out of 318 test outcomes, achieving an accuracy rate of 84%. This means that for the experiments where it predicted a win for a new variant, a significant majority of those predictions turned out to be accurate.

    In contrast, the Qwen3 Next 80B system, which is recognized for its general-purpose capabilities, had a lower accuracy rate, correctly classifying 196 out of 318 outcomes, or about 62%. Despite this difference in performance, both systems provided valuable insights into potential website changes.

    The research highlighted that UX-1 favored new variants in 126 of the tests it evaluated. When these variants were subsequently tested live, 93.7% (or 118 out of 126) of the time, the variants that were predicted to perform better actually won. Only four variants lost, while four results were deemed inconclusive. This strong correlation between the predictions and actual outcomes suggests that using simulated shopper data can significantly enhance decision-making in web experimentation.

    These findings indicate that businesses could benefit from implementing similar predictive models when considering website changes. By relying on simulations that closely mimic real shopper behavior, companies may improve their chances of successful A/B testing outcomes. While the results stem from historical data and simulations rather than live deployments, they point to a promising direction for enhancing how businesses approach website experimentation.

    In summary, the research provides actionable insights for experimentation teams looking to optimize their testing processes. It demonstrates a clear advantage in using specific predictive models, particularly Squoosh UX-1, to inform decisions on which changes to test in a live environment. This could lead to more effective testing strategies and improved website performance over time.

    Academic Abstract

    Experimentation teams must decide which proposed website changes to test with live traffic. We evaluated whether forecasts from simulated shoppers could inform that choice. We screened 412 historical website A/B tests and evaluated four systems on the 318 that ran longer than 21 days. Each system predicted an original win, a variant win, or an inconclusive result. Page roles were masked and recorded outcomes were withheld during inference. Squoosh UX-1, a model fine-tuned on human browsing data, simulated 1,000 shoppers per arm and correctly classified 267 of 318 outcomes (84.0%, 95% Wilson interval 79.5-87.6%, treating experiments as independent). Qwen3 Next 80B, the strongest general-purpose configuration on outcome accuracy, correctly classified 196 of 318 (61.6%). UX-1 favored the variant in 126 experiments. In the live tests, 118 of those variants won (93.7%), four lost, and four had inconclusive results. The remaining 192 experiments included 19 additional variant wins.

    Entities Mentioned

    Companies

    Frequently Asked Questions

    What business problems does this research solve?

    This research addresses the challenge of making informed decisions about website changes by providing accurate forecasts of potential success based on simulated user behavior. This could help businesses reduce the risk of implementing ineffective changes.

    Which industries benefit most from the findings of this research?

    Industries heavily reliant on online presence and user engagement, such as e-commerce, digital marketing, and online services, may benefit most from the insights gained through improved decision-making about website changes.

    What are the practical implementation considerations for businesses looking to utilize this research?

    Businesses may need to consider integrating the forecasting systems into their existing experimentation processes, ensuring that they have the infrastructure to run A/B tests and can collect sufficient data to feed into the predictive models.

    What resources or expertise are needed to effectively implement the findings of this research?

    Organizations may require expertise in data analysis and user experience design, as well as access to historical A/B test data and the technical capabilities to implement and maintain the forecasting systems.

    What are the competitive advantages of using the insights from this research?

    By leveraging accurate predictions of website change outcomes, businesses could gain a competitive edge through more efficient experimentation, leading to quicker optimizations and improved user engagement, ultimately enhancing their online performance.

    Where AI Leaders Stay Informed

    The latest AI intelligence, case studies, and research — delivered to your inbox every week.

    Free to read. Unsubscribe anytime.