Text-to-SQL and data analysis agents: asking databases in natural language

Why it was needed: only people who knew SQL could get data out

Most company data lives in relational databases (systems that store multiple tables linked by keys). Getting the numbers you want out of them requires a query language called SQL. Even a question that looks simple, such as "repurchase rate by region last quarter," needs several tables to be JOINed (linked) and GROUPed BY (grouped and aggregated) before an answer comes out. As a result, business staff often have to file a request with the data team and wait.

Text-to-SQL is a technology that closes this gap. When a person asks a question in natural language, the system converts it into SQL, runs it and returns the result. The BIRD paper paper describes the motivation for this research as "letting non-expert analysts retrieve the information they want from relational databases using natural language."

However, earlier research had been taking tests that were easier than reality. According to the Spider paper paper, most earlier datasets used a single database for both training and testing, and the same SQL appeared in both the training and test sets with only the question wording changed. Some existing benchmarks showed accuracy above 80%, but the paper pointed out that this accuracy was inflated, since models were essentially memorizing question-SQL pairs and picking templates. Conversely, large datasets such as WikiSQL contained only simple single-table queries.

Core idea: a dedicated interpreter for the database

A Text-to-SQL system can be compared to a dedicated interpreter for a database. A good interpreter does not just swap words. First, they learn what the other company's ledger looks like: which tables have which fields and how the tables connect. Next, they work out which field in the ledger the word "sales" in a question refers to, and translate the question into a sentence the ledger understands. If the wording is unclear, they ask back, and they check first that the translated sentence works.

In this analogy, the "ledger structure" is the schema (the blueprint of tables, columns and relationships), and linking words in the question to the schema is schema linking. BIRD paper added one more point: the schema alone is not enough, and the actual values must be understood too. For example, if salaries are stored as strings mixed with "US$" and commas, the special characters have to be removed and the values converted to numbers before an average can be computed. Business knowledge (external knowledge) such as "only owner (OWNER) accounts are eligible for loans" is needed to set conditions correctly.

So today's Text-to-SQL is closer to "understanding the database and then working toward the answer" than to "sentence translation." Since LLMs (large language models) arrived, more systems take the form of AI agents that split this work into multiple steps and use tools along the way.

How it works: from question to answer

LLM-based Text-to-SQL systems generally work in the following order. Implementation details differ from system to system, but the backbone is similar.

First, the system receives the question. Questions unrelated to the database or that cannot be answered are filtered out here, and if the meaning is ambiguous, the system asks back. Second, it performs schema linking. From among the many tables and columns, it picks out only those the question needs. Here it refers to metadata (descriptive information about the data) such as column descriptions, example values and definitions of business terms. This is because putting the entire schema into the prompt would overflow the context window and drive up token costs. Third, it feeds in the selected schema and the question, plus similar question-SQL examples if needed, and has the LLM write the SQL.

Fourth, it validates the SQL before running it. It does a dry run to check for syntax errors, and if an error occurs, it shows the error message back to the LLM and has it fix the query. This "generate-validate-fix" loop has a large effect on accuracy. Fifth, it runs the SQL that passes on the actual database and explains the result table in language people can easily read.

Research differs in which step of this flow it focuses on. As the development history below shows, DIN-SQL focused on question decomposition and self-correction, DAIL-SQL on prompt construction, and CHESS on schema selection and validation.

How the field developed: benchmarks raised the bar, and methods followed

The starting point is Spider paper, released in 2018. Eleven college students spent 1,000 hours building 200 databases (138 domains), 10,181 questions and 5,693 complex SQL queries. The key was how it tested models. It used different databases for training and testing, requiring models to generalize to schemas they had never seen. In this setting, the best model of the time reached only 12.4% exact match accuracy (exact matching, a metric that checks whether the model wrote exactly the same SQL as the reference answer).

In 2023, the arrival of LLMs changed the picture. DIN-SQL paper combined self-correction with a method that breaks SQL generation into smaller sub-problems and feeds their solutions to the LLM in sequence. This raised performance by about 10% over simple few-shot prompting, which shows only a few examples, and lifted execution accuracy on the Spider hidden test set to 85.3. Execution accuracy is a metric that actually runs the SQL and checks whether the result matches the correct answer. The same year, the DAIL-SQL study paper systematically compared prompt engineering techniques such as how the question is represented, how examples are selected and how examples are arranged. Combining these findings, it reached 86.6% execution accuracy on Spider, and it also examined token efficiency and the potential of fine-tuning open-source LLMs.

As Spider scores rose, a question emerged: can LLMs already serve as an interface to databases? BIRD paper, from 2023, answered not yet. With 37 professional domains, 95 databases (33.4 GB in total) and 12,751 question-SQL pairs, BIRD tested large, messy real-world values, external knowledge and SQL execution efficiency. To prevent data leakage, 15 test databases were built separately and kept hidden. Even GPT-4 reached only 54.89% execution accuracy, far below the human score of 92.96%. DIN-SQL scored 55.9% on the BIRD hidden test set.

CHESS paper, from 2024, split the work among four agents: an information retriever that finds relevant values and descriptions, a schema selector that reduces a large schema to a smaller sub-schema, a candidate generator that produces SQL candidates and revises them repeatedly, and a unit tester agent that validates them with natural-language unit tests. Schema selection alone raised accuracy by about 2% and cut LLM token usage to one-fifth, and the system scored 71.10% on the BIRD test set. The same year, Spider 2.0 paper moved the test to the enterprise setting. It built 632 workflow problems on real databases stored in cloud data warehouses such as BigQuery and Snowflake, some with more than 1,000 columns. Solving the problems requires consulting metadata, documentation on SQL dialects (the slightly different SQL syntax each database uses) and even project code, and sometimes writing SQL longer than 100 lines. A code agent based on o1-preview solved 91.2% on Spider 1.0 and 73.0% on BIRD, but only 21.3% on Spider 2.0.

Where it is heading: data agents that ask back, stop and act

In 2026, research attention is shifting from "does it write correct SQL?" to "can it be trusted?" and "can it do work beyond SQL?"

The first line of work is ambiguity. ARCS paper argues that ambiguity in questions becomes a main source of errors in real deployments. Instead of asking clarifying questions through open-ended dialogue, it proposes "structured ambiguity resolution," which resolves ambiguity through explicit, constrained interaction. The study built a benchmark by annotating ambiguities that arose naturally in real databases, along with their possible interpretations and the SQL for each interpretation. On it, even gpt-6-sol reached only 51% end-to-end execution accuracy, and open-source models did not exceed 27%.

The second line is knowing when to stop. The Sentinel study paper, which deals with querying medical records, adds a confidence-assessment layer that analyzes the agent's reasoning, code and database outputs, and decides whether to stop at three points. Before execution it rejects unanswerable questions, during execution it cuts off unpromising Trajectories, and at the end it withholds answers that are hard to trust. On the EHRSQL benchmark, the share of questions answered fell from 84% to 55%, but the accuracy of the answers given rose from 54% to 69%.

The third line is expansion into data agents. Argo-Bench paper points out that existing Text-to-SQL benchmarks evaluate only query generation, and that an audit found frequent errors even in their answer keys. The study built 210 data science and analytics tasks on an ERP (enterprise resource planning) warehouse with 235 tables and 7.5 billion rows. Agents go as far as taking actions such as suspending fraudulent accounts or allocating incentive budgets for delivery drivers, and they are scored on the outcomes those actions produce in a simulator. Among 14 models, even the strongest scored 95 or above on only 34.8% of tasks.

Limitations and open issues

The biggest limitation is that "it ran" is not the same as "it was right." SQL with no syntax problems can still misread the meaning of a question and return numbers that look plausible but are wrong. Because the results come back as a clean table, users find it hard to be suspicious. As ARCS paper notes, ambiguity is subtle and varies by domain and data, so it makes systems quietly drift away from what the user intended. In practice, therefore, many deployments show the generated SQL together with the assumptions the system made, and add human-in-the-loop checks in which people verify important numbers.

Second, evaluation itself is shaky. Benchmark scores do not directly reflect performance on real work. Agents that scored high on Spider 1.0 and BIRD dropped sharply on Spider 2.0 paper, and Argo-Bench paper flagged errors in the answer keys of existing benchmarks. The metrics have weaknesses too. Exact match marks SQL that is written differently but produces the same result as wrong, and execution accuracy can mark SQL that only happens to produce the same result as correct.

Third, there are operational issues. As BIRD paper reflects by including a separate efficiency metric, on large databases even correct SQL is less useful if it is slow. And because the setup runs SQL written by an LLM as is, basic safeguards such as read-only permissions, execution time limits and access controls on sensitive columns must be in place regardless of model performance. When column names are hard to decipher or definitions of business terms are scattered across many places, any model will struggle. That is why work that organizes the "meaning" of data, such as a data catalog or semantic layer, is the foundation of accuracy.

Further study

If you are new to the field, start with the introduction of Spider paper. Why models must be "tested on databases they have never seen" is the core of how the field defines its problem. Next, the three examples in Figure 1 of BIRD paper (messy values, external knowledge, execution efficiency) give a sense of why real-world data is hard.

For methods, it helps to read, in order, DIN-SQL paper for "decompose, solve and self-correct," the DAIL-SQL study paper for the effect of each prompt component, and CHESS paper for a division of labor among agents. If you have enterprise settings in mind, check what Spider 2.0 paper demands beyond earlier benchmarks.

For recent trends, reading ARCS paper on ambiguity, the Sentinel study paper on confidence and deciding when to stop, and Argo-Bench paper on evaluating data agents beyond SQL covers most of the main issues of 2026.