Paper review · October 19, 2026
PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems
The authors built a benchmark that pairs 1,800 Text-to-SQL queries with false premises with their 1,800 original queries. They showed that LLMs take flawed questions at face value by default, and that when strongly instructed, they reject even valid queries.
Problem
Existing Text-to-SQL (NL2SQL) research assumes that user questions are always valid and answerable. As a result, models produce SQL that runs but is semantically wrong, even for questions with contradictory conditions or questions about elements not in the schema. They also often quietly fill in unsupported assumptions.
Method
The authors define the ability to notice a flawed question and withhold SQL generation as 'premise critique.' They divide premise errors into 4 main categories (expression errors, premise contradictions, missing premises, and out-of-scope requests) and 9 subtypes. They modified Spider 1.0 and BIRD queries with GPT-5.1 to create flawed queries (positives), ran automatic verification with Gemini-2.5-Pro followed by human review, and paired them with the original queries (negatives). There are 3 prompts: Base (ordinary SQL generation), Implicit (tells the model the question may contain errors), and Explicit (states that the question is flawed). Positive samples were scored on degree of SQL abstention, flaw detection (DET) and type identification (TYPE); negative samples on SQL generation rate (GEN) and validity judgment (VAL). They evaluated 172,800 responses from 16 recent LLMs.
Results
- In the Base setting, the aggregate flaw detection rate (DET) across the 16 models was only 0.319, and the SQL abstention score only 0.177. The SQL generation rate (GEN) on original queries was 0.988. Even the top model, Claude Opus 5.5, had a DET of 0.492, missing more than half of the flaws.
- In the Implicit setting, aggregate DET rose to 0.672, while original-query GEN stayed at 0.903 and VAL at 0.826. However, variation was large. DeepSeek V3.2 had the highest DET at 0.898, but its original-query GEN was only 0.455.
- In the Explicit setting, aggregate DET rose as high as 0.931. However, GEN on original queries fell to 0.523 and VAL to 0.321, meaning models rejected many valid queries as well.
- The most extreme case was Qwen3.5-397B. Its DET on flawed queries was 0.968, but its original-query GEN was 0.004 and VAL 0.005. By contrast, Gemini 3.1 Pro Preview was less affected, with original-query GEN of 0.941 and VAL of 0.704.
- The authors report that strong instructions raise apparent critique performance but do not produce consistent, evidence-based premise judgments.
Limitations
Because the flawed queries were generated with GPT-5.1 and verified by an LLM and humans, their error distribution may differ from errors real users make. There are only two source datasets, Spider 1.0 and BIRD. The evaluation compared only 3 prompts at temperature 0 and does not propose a training method to improve this ability. GEN measures whether SQL is generated, not whether the generated SQL is correct. Execution-based verification is in the appendix. The provided text was cut off partway, so later analyses are not covered.
Why it matters
Natural-language database query tools used at work can return plausible SQL and results even when the question is wrong. This study shows that this risk is real and large. It also shows that a prompt saying 'reject if there is an error' on its own creates the opposite problem of rejecting valid queries. In practice, flaw detection rate and valid-query handling rate should be measured together.
Key terms
- Text-to-SQL (NL2SQL)
- The task of taking a natural-language question and a database schema and producing executable SQL.
- Premise critique
- The ability to check whether the assumptions in a question hold in the database context and to find flaws. It includes withholding output instead of producing wrong SQL when a flaw is found.
- Abstention
- The behavior of stopping generation instead of forcing an answer to a question that cannot be answered or is flawed. In this paper, it corresponds to outputting null as the sql value.
- Positive/negative pair evaluation
- An evaluation method that tests flawed queries (positives) together with their originals (negatives). It distinguishes whether a model rejects based on evidence or rejects unconditionally because it was told to.
Original
Authors: Shuo Lin, Jinzhe Li, Chenlu Guo, Yanhan Li, Yuan Wu, Yi Chang · Posted 2026-10-15 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.