ARCS: Precise Text-to-SQL Through Structured Disambiguation

Problem

Most Text-to-SQL benchmarks consist of unambiguous, overly detailed questions. Ambiguity in real user questions (e.g., whether 'revenue' includes refunds) silently produces wrong results. Existing approaches that ask clarifying questions through dialogue are inefficient and place a heavy burden on users.

Method

The authors propose 'structured disambiguation'. It finds the ambiguous points in a question and lets the user choose: via multiple choice if the interpretations are finite, or via a slider if they are infinite, such as a numeric threshold. It also generates parameterized SQL matching the selected combination. Ambiguity is classified along two orthogonal dimensions: a language axis (semantic, syntactic) and a database axis (column, table, value, computation). Five BIRD databases and one Spider 2.0 database were used without modification. Question candidates were generated with gpt-4.1, and the authors manually annotated every ambiguous point, interpretation and SQL query (about 2.5 hours per question). Evaluation was split into an end-to-end setting using an LLM user simulator and a fine-grained setting that separates disambiguation from SQL generation.

Results

• The authors reported that gpt-6-sol reached only 51% end-to-end execution accuracy on ambiguous questions, and no open-source model exceeded 27% (e.g., gptoss-120b at 26.05%).
• For every model, end-to-end accuracy was lower than execution accuracy when given the correct interpretation (EX_disambiguated). For example, claude-opus-4.5 dropped from 67.85% to 38.59%.
• The authors reported that frontier LLMs with similar Text-to-SQL ability differ greatly in disambiguation performance. They suggest that disambiguation may have received less attention as a training objective.

Limitations

With only 6 databases, domain coverage is limited. The scale is about 300 instances sampled from 101 questions, and the task count is given inconsistently in the text as both 311 and 331. End-to-end evaluation relies on an LLM user simulator, which may differ from real users. Question candidates were generated by gpt-4.1, and annotation was done by the authors themselves.

Why it matters

This has implications for enterprise data analytics services that connect Text-to-SQL to internal databases. If ambiguous questions are passed through as-is, the numbers silently come out wrong. It supports designs that confirm user intent through lightweight UI such as multiple-choice selection.

Key terms

Text-to-SQL

The task of converting a natural-language question into an SQL query for a database.

Structured disambiguation

Instead of open-ended clarifying questions, this approach shows options and sliders for each ambiguous point and lets the user pick to confirm their intent.

Execution accuracy (EX)

A metric that checks whether the execution result of the generated SQL matches that of the reference SQL.

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Project
• Hugging Face

Authors: Yihao Hu, Yanlin Feng, Naoki Otani, Nikita Bhutani · Posted 2026-10-07 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.