LLM evaluation: leaked exam papers, wavering graders and the practical test of real work

Why evaluation has become hard

The basic framework for LLM evaluation is the benchmark, a set of questions with fixed correct answers. The model answers the questions and the share of correct answers is counted. When answers are free-form, overlap metrics such as BLEU and ROUGE are widely used. They count how many word fragments the model's answer shares with a reference answer. For example, a medical QA comparison study paper compared five models on 38,000 medical questions from the iCliniq dataset. It used a zero-shot approach, asking directly without any fine-tuning, and graded with BLEU and ROUGE.

But as models began writing freely like people, this approach started to break down. According to a study on judging biomedical relation extraction paper, LLMs often produce synonyms or abbreviations of the correct answer. That makes traditional automatic metrics hard to trust. Human grading is more reliable, but it is expensive and slow, so it is hard to use every time in a real service. This is why LLM-as-a-Judge, which hands grading to another LLM, has spread.

The problem is not only grading. The exam itself may have been mixed into the training data (data contamination). Exam scores may also be far removed from real-world job performance. This note covers these three cracks in turn: contamination, judging and real work.

Core idea: the exam paper, the grader and the practical test

LLM evaluation is easier to understand by comparing it to a certification exam. If the exam paper leaked in advance, a high score shows memorization, not ability. If the person standing in as grader is easily swayed by the tone or length of an answer, the score cannot be trusted. And someone who does well on the written exam may still touch the wrong thing in the practical test.

So when reading evaluation results, you need to ask three questions. First, is this score really the result of solving problems the model has never seen? Second, who graded it, with what settings, and has that grader been validated? Third, how closely does this test resemble the work I want to hand off?

There are trade-offs among the three questions. Tests with fixed answers are cheap and give the same result however many times you run them, but they are far from reality. Tests that resemble real work are realistic, but they are hard to build and hard to grade against a single correct answer. LLM judging fills that gap, but it raises a new question: can the grader be trusted? Recent research is looking for the right balance in these trade-offs.

How it works: the flow of a single evaluation

A single evaluation generally runs in the following order. First, prepare a question set and check that the questions do not overlap with the training data or retrieval sources. Next, run the model and collect its answers. For agents that use tools, also keep an execution log (trajectory) containing every intermediate action along with the final answer. Then grade, and finally aggregate and interpret the results.

There are three main grading methods. The first compares answers against correct answers or rules. It is cheap and reproducible, but it marks an answer wrong if the wording differs even slightly. The second is LLM-as-a-Judge. You put the grading criteria, the question and the model's answer into a judge prompt and get back a score or a win/loss verdict. Scoring a single answer is called pointwise, and choosing the better of two answers is called pairwise. RobustJudge paper tested both. The third is human evaluation, which is the most reliable but also the most expensive.

LLM judging has several knobs that change the result: which model to use as the judge, how to write the judge prompt, and what temperature to set. Temperature is a value that controls how much randomness is allowed when generating an answer. The studies introduced below show that each of these knobs can shift the verdicts. That is why, before using a judge, you need a step that compares it with a sample graded directly by people and checks the agreement rate.

How the field developed: research digging into the three cracks one by one

Start with contamination detection. LogitTrace paper points out that even on hard benchmarks such as AIME and Math500, contamination can make memorized solutions look like genuine reasoning. Earlier detection methods relied mostly on surface information: how much the text overlaps, whether the model continues the rest of a question verbatim when given the first part, and how confident it is in the final answer. As a result, their performance dropped when a question was simply reworded. LogitTrace instead watches how the answer the model prefers (the logits) changes across each internal layer. On contaminated questions, the model settles on an answer early; on clean questions, it builds up evidence gradually. Using this difference as a signal, even a lightweight classifier could tell the two cases apart across multiple models and input variations. And when specific questions were deliberately trained repeatedly with LoRA, a similar pattern appeared.

The weaknesses of judges showed up in two forms. One is a format problem. The biomedical relation extraction judging study paper used 8 LLM judges to grade responses from 5 other LLMs on 3 datasets. Judging accuracy was generally below 50%. The main cause was that the relations extracted by the LLMs did not follow the required format. When responses were received in a structured output format, judging performance rose by about 15% on average. The study also proposed a domain adaptation technique for transferring knowledge between datasets, and released 36k judgments made by humans and LLMs.

The other is a robustness problem. RobustJudge paper tested 15 attack techniques and 8 defense strategies on 13 models. Whether pointwise or pairwise, LLM judges remained vulnerable to attacks. Composite attacks that combined several techniques were the strongest and transferred well to other models. The judge prompt template and the choice of model also affected robustness. The judge temperature study paper ran more than 1 million evaluations to measure the effect of temperature. Lower temperatures made verdicts more consistent and reduced errors, especially for smaller models. But low temperature also narrowed the range of reasoning the model considered. On ambiguous cases where it is hard to tell which answer is better, changing the method improved agreement: for example, sampling several times at high temperature and taking a majority vote, or reasoning at high temperature and making only the final decision at low temperature. Verdicts on clear-cut cases did not get worse, but inference costs went up.

On the real-work side, EconEvals paper released an open-source evaluation suite that measures capability at the level of tasks and occupations in the US labor market. Where possible, it builds questions from real user queries and fills gaps with synthetic data. According to the study, OpenAI's GDPval, the previous state of the art, covers 5% of US occupations. EconEvals states that it covers a broader range at 500 times lower cost. The study estimated that in 47% of occupations, the time spent on more than half of the tasks could be cut substantially. But for 79% of those tasks, actual Claude usage was low. The researchers named privacy and proprietary in-house systems as the main bottlenecks.

Where things are heading: from average scores to conditions and process

The first trend is that contamination detection is widening into an audit of leakage across the entire evaluation procedure. The rare disease diagnosis audit study paper rebuilt and tested state-of-the-art agent frameworks using more than 60,000 cases drawn from public and independent clinical cohorts. When the space from which diagnosis candidates are chosen was not properly separated from the evaluation data, performance was inflated. With strict isolation between datasets, average Recall@1 (the share of cases where the first candidate is correct) fell from 54.3% to 40.3%. Supervised classifiers likewise averaged 42.9% accuracy in internal evaluation but only 16.7% on external cohorts. The researchers also showed quantitatively that LLMs infer diagnoses backward from post-diagnosis treatment cues in clinical notes. They also found that performance swings when the time frame of the knowledge base is misaligned. Their conclusion is that current evaluation methods may reflect data leakage more than medical competence.

The second trend is breaking down judge behavior by condition, such as judge, task and language. The multilingual judging study paper tested 9 prompting strategies, such as CoT and role assignment. Combining these with 4 open-weight judges, 27 languages, 4 tasks and 13 datasets, it collected more than 1 million scores and validated them against human judgments. Strategies known to work well on English benchmarks did not carry over to other conditions. The choice of judge model had a bigger effect on results than the prompting strategy. The same strategy sometimes had opposite effects depending on the judge. Judging was unstable in low-resource languages, and this problem was hidden when looking only at the overall average. The researchers recommend validating prompting strategies directly in the evaluation setting where they will actually be used.

The third trend is evaluating agents not only on outcomes but also on process and side effects. SABench paper consists of 201 email, calendar and ticketing tasks. It combines 4 types of task-state structure with 4 types of traps, and attaches an oracle (an automatic judge) that checks each task's execution log against rules. The oracle checks both whether the goal was achieved and whether permissions were respected. Running 11 frontier LLMs across 2,211 model-task pairs, collateral damage occurred in 15.2% of all runs: writing to the wrong target, broadening the scope of a request, contacting unintended recipients and so on. The violation rate was 38.8% on ambiguous tasks and averaged 8.4% for the other traps. Even among runs that achieved the goal, 12.8% performed unauthorized writes. Among models with similar goal-achievement rates, collateral damage rates differed by up to 4.7 times. This means task success rate alone cannot tell you whether a model uses tools safely.

Limitations and open issues: what to weigh in practice

LLM judging fits best when there are so many open-ended answers that people cannot grade them all. Conversely, when the answer format is clear, rule-based grading is cheaper and more reproducible. Be careful in settings where the system being evaluated might try to fool the judge. As RobustJudge paper showed, judges are vulnerable to attacks, and the defenses tested were not complete. Either way, the starting point is to first build a sample graded directly by people and measure how well the judge agrees with it.

There are five common mistakes. First, using a judge prompt known to work well on English benchmarks as-is for other languages or other judges. The multilingual judging study paper showed that the effect can reverse. Second, setting the judge temperature by gut feeling. The rule from the judge temperature study paper is to default to a low temperature for stability, and to sample multiple times and deliberate only on hard-to-judge cases. Third, reading public benchmark scores directly as performance on your own work. Contamination or candidate-space leakage can inflate scores paperpaper. Fourth, looking only at a single overall average and missing weaknesses by language or task type. Fifth, looking only at an agent's completion rate without checking for actions outside its permissions paper.

Some issues remain open. Methods that inspect per-layer logits, like LogitTrace paper, require access to the model's internal values. So they are hard to apply as-is to models whose internals cannot be inspected. Real-work evaluation must keep trading off coverage against cost. The bottlenecks EconEvals paper identified, privacy and proprietary systems, also mean that the more internal a company's work is, the less public evaluations can measure it. In the end, you have to build your own evaluation set for in-house work. And as long as you use a judge, the question "who validates the judge?" remains. Human validation can be reduced but not eliminated.

Further study

To start with LLM judging, the biomedical relation extraction judging study paper is a good first read. It shows concretely why judgments go wrong and what improves when you change the format. Next, use RobustJudge paper to see the full map of attacks and defenses, and the judge temperature study paper to learn settings rules you can apply right away. Finally, reading the multilingual judging study paper rounds things out by showing that those rules vary with conditions.

For contamination and leakage, get the principles of detection from LogitTrace paper, then read the rare disease diagnosis audit paper. That gives a sense of how much small gaps in evaluation design can change the numbers. For real-work evaluation, look at EconEvals paper for the big picture at the level of tasks and occupations, and SABench paper for how to grade agent side effects from execution logs. If you need an example of the traditional approach of grading with overlap metrics, the medical QA comparison study paper is short and easy to read.