Paper review · November 4, 2026

NitiBench+: A Time-Aware Thai Legal Question-Answering Benchmark Across 17 Areas of Law

NitiBench+: A Temporal Thai Legal Question Answering Benchmark Across 17 Domains

Data & evaluation Language models Agents Legal & professional services Intermediate cs.CL

The authors built a Thai legal QA benchmark that includes both the statutory text in force on the date of each ruling and the current text, showed that model scores drop when models read the current text, and released a leaderboard of 26 models.

Problem

Courts apply the law in force on the day of the ruling, but legal QA benchmarks grade against current law. Of the 53 legal benchmarks the authors surveyed, none graded the version of the statute. Even NitiBench, the authors' earlier benchmark, saw 2.3% of its citations stop matching the current statutes within two years.

Method

From rulings of Thailand's Supreme Court, Constitutional Court and Supreme Administrative Court, the authors built 1,670 questions (4,071 sub-questions) across 17 areas of law. Each question gives the facts and asks for the conclusion, the reasoning and the applicable provisions. Reference answers are taken verbatim from the rulings. For every cited provision, both the text in force on the ruling date and the current text are included. The retrieval corpus contains every dated version of every Thai statutory provision (106,041 provisions, 394,943 texts). Grading has two parts. One is a Bar-standard score (0–10) based on the Thai Bar Association's grading criteria, using human graders and a calibrated LLM judge. The other is the overlap between the provisions a model cites and those the court cited (Citation F1). Part I measured the effect of switching provision versions using 13 open-weight models. Part II evaluated 26 models in three settings on a 500-question private test set stratified by domain. The three settings are answering from memory alone, answering with the court-cited provisions provided, and answering as an agent that searches the statutes in force.

Results

An analysis of 156,650 rulings from four courts found that the share of rulings whose cited law was later amended or repealed was 67.1% for the Supreme Court, 62.4% for the Constitutional Court, 42.6% for the Administrative Court and 23.3% for the patent and specialized appeals court.
Even when given the correct provisions, models lost 0.25 points on the Bar-standard score when they read the current text of amended provisions. When repealed provisions could not be found under current law, scores fell by 0.79 points.
The score drop was 0.45 points on the 428 questions where the rule itself changed, and there was no drop on the 331 questions where only the wording changed. Citation F1 did not capture this difference.
The top model on the leaderboard scored 7.78 out of 10, and the top open-weight model scored 6.39.
Only five open-weight models recalled any Thai law at all without a search tool. With a search tool, 20 of the 26 models improved and 6 got worse.

Limitations

The benchmark is limited to Thai law and the Thai language. Grading relies on human graders and a calibrated LLM judge. The Part I version-effect analysis used only 13 open-weight models. The leaderboard test set has 500 questions (29–30 per area), which leaves little statistical room for per-area comparisons. The authors state that model outputs are not legal advice.

Why it matters

When building legal RAG systems or legal agents, checking only that provision numbers match can miss answers that are wrong for the point in time. The paper shows with numbers that versioning by ruling date is essential. It is also useful as a public leaderboard showing the limits of current models in non-English legal domains.

Key terms

Statutory temporal drift: The situation where a provision that a ruling or question relied on is later amended or repealed, so it no longer matches the current text.
Bar-standard score: A score based on the Thai Bar Association exam grading criteria (0–10) that evaluates not only the conclusion but also the legal reasoning.
Citation F1: A metric that uses precision and recall to measure how much the provisions a model cites overlap with the provisions the court actually relied on. It does not account for differences between provision versions.

Original

Authors: Pawitsapak Akarajaradwong, Sarana Nutanong, Chompakorn Chaksangchaichot · Posted 2026-11-02 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.