AI Trend Research

← AI glossary

AI glossary · Advanced · Data & evaluation

LLM-as-a-Judge

LLM-as-a-Judge is an evaluation method in which, instead of people, an LLM (large language model) scores another AI's answers or picks the better of two answers.

About this term
Level
Advanced · Specialist terms that need the inner workings
Added
2026-10-08
In plain words It is like a teacher who cannot grade a huge stack of exams alone and hands first-pass grading to a teaching assistant who knows the grading rubric well. However, this assistant also makes mistakes and has habits, so the teacher needs to recheck from time to time.

How it works

First, the AI model being evaluated is given a question and its answer is collected. Next, the question, the answer and the evaluation criteria are put together into a prompt for an LLM acting as the 'judge.' The judge LLM assigns scores based on criteria such as accuracy, helpfulness and safety, or compares two answers and picks the better one. Having it also write the reasons for its verdict makes it easier for people to check why a given score came out. To see whether the results can be trusted, some verdicts are cross-checked against human evaluations.

How an LLM judge scoresQuestion → Model being evaluated; Model being evaluated → Answer; Answer → Judge LLM; Evaluation criteria → Judge LLM; Judge LLM → Score · verdict; Human evaluation → Score · verdict(Cross-check)QuestionModel beingevaluatedAnswerEvaluationcriteriaAccuracy · helpfulness· safetyJudge LLMScore · verdict+ reasons for verdictHuman evaluationCross-check a sampleCross-check
How an LLM judge scores When the evaluated model's answer and the evaluation criteria are fed into the judge LLM, it produces a score or verdict. Some verdicts are cross-checked against human evaluations to confirm they are reliable.

Why it matters

Tasks without a single correct answer, such as summarization, writing and conversation, are hard to grade with multiple-choice benchmarks. Having people evaluate every answer is accurate but takes a lot of time and money. Using an LLM judge makes it possible to evaluate large numbers of answers quickly, cheaply and by the same criteria. That is why it is widely used for checkpoints during model development, monitoring service quality, and creating preference data for training.

Things to know

Judge LLMs have biases too. They are known to favor answers that appear first or are longer, and to rate answers written by models similar to themselves more highly. On problems requiring specialized knowledge the judge itself lacks, it may mark wrong answers as correct. To reduce these problems, evaluators swap the order of answers and evaluate twice, write very specific criteria, or combine verdicts from multiple judges. In the end, it cannot fully replace human checking, and it is safest to use it as a supporting tool.

Example

A company running a chatbot service collects new answers to hundreds of sample questions every time it modifies its prompt. It then has a judge LLM compare them with the previous answers using criteria such as 'Did it answer the question accurately? Did it avoid making up unsupported content?' If people review only the items whose scores dropped, they can catch quality changes much faster than if they read everything themselves.

Related terms

Benchmark LLM Prompt RLHF Sycophancy Data contamination