AI glossary · Basic · Data & evaluation
Benchmark
A benchmark is a standard set of test questions and a scoring method designed to compare AI models' abilities under the same conditions. Each model solves the same problems, and their skill is compared by score.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Foundations & applications
- Fields
- Data & evaluation
- Added
- 2026-10-08
How it works
A benchmark usually consists of a set of problems, answers or scoring criteria, and a method for calculating scores. Some are tests with clear answers, such as math problems, coding tasks or common-sense questions, while others have humans or another AI evaluate answer quality. Recently, more benchmarks check whether AI agents can complete real tasks end to end across multiple steps. Because the same problems and scoring criteria are used, models from different companies can be compared side by side.
Why it matters
To verify claims that an AI model has 'gotten smarter', a common yardstick is needed. Researchers use benchmarks to check whether a new method actually improves things, and companies refer to them when choosing which model to use. New benchmarks reveal what current AI cannot do, which also influences research directions. That is why scores on several benchmarks are often published together when a model is announced.
Things to know
If benchmark questions get mixed into training data, a model can 'memorize' them and inflate its score; this is called data contamination. When models get close to perfect scores, the benchmark can no longer tell them apart and is said to be 'saturated'. Also, if a model is tuned only to raise scores, it may perform worse than expected in real use. So it is best to look at scores across several benchmarks together with real usage experience.
Example
'TasteVal', covered on this site, is a benchmark comparing AI's sense for experimental research with that of human experts, and 'LMBuild' evaluates whether LLM agents create structures that can actually be assembled and that work. Like these, benchmarks today are expanding beyond getting the right answer to measuring research judgment and the ability to produce results that actually work.
Where it came up on this site
- Papers From evidence to action: Where do tool-using agents fail? 2026-10-08
- Papers LMBuild: Evaluating LLM agents that build structures that can actually be assembled and work 2026-10-07
- briefing AWS releases 'aws-ai-ml', a SageMaker inference optimization skill for coding agents 2026-10-06
- briefing TasteVal: A benchmark comparing AI's sense for experimental research with human experts 2026-10-06