AI glossary · Beginner · MLOps & LLMOps · Data & evaluation
A/B testing
A/B testing is an experimental method that randomly splits users into two groups, shows one group the existing version (A) and the other a new version (B), and then compares with real data which one produces better results.
About this term
- Level
- Beginner · Core ideas for first-time readers
- Category
- Basics
- Fields
- MLOps & LLMOps, Data & evaluation
- Added
- 2026-10-08
How it works
First, you define the hypothesis to test, such as "the new recommendation method will increase clicks," and the metric that will determine success. Next, you randomly assign users to group A and group B so that the two groups are as similar as possible except for the version. You collect each group's metrics (click-through rate, purchase rate, satisfaction and so on) over a sufficient period. Finally, you check statistically whether the difference between the groups is at a level that could arise by chance, to decide whether it is a "meaningful difference."
A/B testing steps
- 1Set hypothesis and metricsWhat should improve, and how to measure it
- 2Split randomlyKeep groups A and B similar
- 3Collect dataOver a sufficient period
- 4Decide with statisticsCheck whether the difference is chance
Why it matters
It lets you make decisions based on real user responses instead of gut feeling in a meeting room or the opinion of whoever is loudest. In AI services, it is widely used to check whether a new model version, a changed prompt or a different recommendation algorithm is really better. A high score on a prepared test set (offline evaluation) may not translate into a better experience for real users, so it is important as a final check before and after deployment.
Things to keep in mind
With too few users or too short a period, it is easy to mistake a chance difference for a real effect. If you watch many metrics at once, one of them may look good by chance, so it is best to define the key metric in advance. A new feature may get a good response at first because it is novel and then cool off, so allow a generous test period. Also, because one group of users may be inconvenienced, test risky changes on a small share of users first.
Example
Search, shopping and video services show a new version to only some users when they change the order of recommendation lists or the screen layout, and compare clicks and dwell time. Teams running generative AI chatbots also use A/B testing: they apply a new model or a revised system prompt to only some conversations, check whether answer satisfaction (thumbs up/down) and the rate of follow-up re-asking improve, and then roll it out to everyone.