Agents and tool use: how models went from talking to working

Why it was needed: models that talk well but get calculations wrong

LLMs (large language models) generate text by predicting the next token (a piece of text). This approach works well for translation, summarization and conversation, but as the Toolformer paper pointed out, they make mistakes on tasks such as arithmetic and fact lookup that much smaller, simpler programs handle well. They do not know about recent events, and they also produce hallucinations, plausible-sounding falsehoods.

Meanwhile, having models write out their reasoning step by step (chain-of-thought) improved reasoning ability, but the ReAct paper described this as a 'closed black box.' Because the model reasons only from the knowledge in its own head, it cannot check against the outside world, and if an incorrect fact slips in midway, the error carries through to the end.

Conversely, some research had models only plan actions, but that work lacked the 'thinking' needed to reason about high-level goals or keep track of progress. This gap, with thinking and acting studied separately, is the starting point for research on agents and tool use.

Core idea: a new hire with a notebook

An AI agent is an LLM system that, given a goal, breaks it into steps on its own and uses external tools to finish the job. Here, tools are programs outside the model that return accurate results, such as calculators, search engines, the Wikipedia API, code executors and web browsers.

Think of a new employee. Given a task, they write in a notebook, 'First I should check last quarter's sales' (thought), query the company system (action), look at the resulting numbers (observation), and write again, 'Lower than expected; let me dig further into the cause.' They do not make up answers from memory alone; they look up what they do not know and revise their plan based on what they find.

ReAct carried this notebook practice directly over to LLMs. The ReAct paper described it as 'reason to act, and act to reason.' Thinking helps the model make plans and handle exceptions, while acting pulls in outside information and keeps the thinking grounded in facts.

How it works: the thought–action–observation loop

Step 1: receive the question. The agent reads the instructions and a few examples in the prompt to work out what it needs to do. ReAct also worked when given only one or two examples.

Step 2: write down a 'thought'. The agent writes what it knows now, what is missing, and what it will do next. Step 3: produce an 'action'. This means calling a tool in a fixed format, for example 'search Wikipedia for X'. Toolformer wrapped such calls in special markers, inserted them into the text, and appended the result returned by the tool at the same spot.

Step 4: the system actually runs the tool and returns the result as an 'observation'. The agent reads this observation and goes back to step 2 to continue thinking. When it judges that it has gathered enough evidence, it gives the final answer.

An extra mechanism can be added on top. In Reflexion, when an attempt fails, the agent looks at the feedback, writes a reflection on 'what went wrong', stores it in memory (episodic memory), and reads that reflection on the next attempt. This lets the model learn through text alone, without changing its parameters.

How the field developed: what each foundational study introduced

ReAct (2022) had the model alternate between thoughts and actions in a single stream. On question answering (HotpotQA) and fact verification (Fever), using only a simple Wikipedia API reduced the hallucination and error-propagation problems of the chain-of-thought approach. On a text game (ALFWorld) and shopping-website navigation (WebShop), with only one or two examples it raised success rates over imitation-learning and reinforcement-learning methods by 34 and 10 percentage points in absolute terms, respectively. The authors also cited as a strength that people can read the thought trace and check the reasoning behind decisions. However, the paper reported that a combination of ReAct and chain-of-thought, which uses internal knowledge, performed best overall.

Toolformer (2023) let the model learn by itself 'when to call which tool, with what input'. Given only a few human-written examples per tool, the model inserted candidate tool calls into a large body of text; the calls were then actually executed, only those that helped next-token prediction were kept, and the model was fine-tuned on that data. It covered a calculator, question answering, search, translation and a calendar, and a 6.7-billion-parameter model based on GPT-J outperformed the much larger GPT-3 on several tasks.

Reflexion (2023) changed how agents learn from trial and error. Instead of using many samples and costly fine-tuning as in conventional reinforcement learning, it had the agent reflect on failures in language. It reported pass@1 of 91% on the coding benchmark HumanEval, exceeding the 80% of GPT-4 at the time.

CodeAct (2024) changed the format of actions. Agents usually call tools in JSON or a fixed text format, which makes it hard to go beyond a predefined set of tools or to combine several tools. CodeAct unified actions as executable Python code, so a single block of code can chain several tools and the agent can revise it while looking at the execution results. An analysis of 17 LLMs showed success rates up to 20% higher than existing approaches, and the authors also released 7,000 multi-turn interaction examples (CodeActInstruct). A survey on LLM-based autonomous agents (2023) that summarized this line of work brought agent construction methods together into one unified framework and also reviewed applications and evaluation methods.

Real-world evaluation: much harder than expected

An agent that uses tools well on simple tasks does not necessarily do well on real work. WebArena (2023) built reproducible environments of working websites in four domains: e-commerce, community forums, collaborative software development and content management. When given the long, complex tasks people commonly do on the internet, the best GPT-4-based agent achieved a success rate of 14.41%, far below humans (78.24%).

SWE-bench (2023) collected 2,294 problems from real GitHub issues and the commits that resolved them in 12 popular Python repositories. The model has to look at the whole codebase, edit several functions, classes and files together, interact with the execution environment and handle very long contexts. Even Claude 2, the best model at the time, resolved only 1.96%.

Both evaluations show the same thing. Getting the tool-call format right and correctly completing a long process from start to finish are entirely different problems. This is the background for research interest later shifting to 'how reliably does the agent work'.

Where things are heading: agents that know when to doubt their tools

First, when a tool gives a wrong answer. The 'When Tools Lie' study, published in October 2026, secretly replaced the tool results of a math-solving agent with plausible wrong answers. On 31 problems, without a verification mechanism accuracy fell from 100% to 72.4%; adding 'forced reflection', which requires the agent to re-check within the same context, restored it to 100%. When a verification tool was offered only as an option, it helped only when the model actually called it. The conclusion is that having a means of verification and having a rule that makes the agent verify are two separate things.

Second, when the agent acts without evidence. The 'From Evidence to Action' study built SafeActBench, consisting of 656 cases, and traced what the agent had confirmed before acting. Failures often began before the execution stage: the agent stopped before finishing its investigation, or acted before the necessary evidence was secured. A single action taken after securing evidence was generally reliable, but tasks involving a chain of actions revealed more problems such as missing preconditions or stopping partway.

Third, when the agent does not know when to stop. In the 'Judged Useless, Queried Anyway' study, seven agents correctly judged 97–100% of results from a broken search source as 'useless', yet most did not stop based on that judgment. Even when the rule was given in the prompt, they followed it only partly. When the harness forced the agent to answer after five consecutive useless results, success rates rose for all models. All three studies say that enforcing verification and stopping through external structure is more effective than relying on the model's judgment alone.

Limitations and open issues

The first issue is trust. Agents tend to accept tool results as 'the right answer'. If a tool is silently wrong, the error spreads through all later reasoning. The error propagation ReAct tried to reduce now enters from the tool side.

The next issue is evaluation. Success rates on short tasks may be high, but they dropped sharply on long real-world tasks such as WebArena and SWE-bench. And as recent studies show, judging well whether 'this action is right' in a static setting is different from executing correctly while actually interacting. A correct result also does not guarantee that the process was grounded in evidence.

Finally, there remains the issue of design choices. Whether to restrict tool calls to a fixed format or open them up freely through code as CodeAct does is a balance between flexibility and control. There is also no settled answer yet on whether reflection or verification should be left to the model's discretion or enforced by rules. Enforcing them is more stable but can add unnecessary checks.

Further reading

If you are new to this, start with ReAct paper. The thought–action–observation loop, the basic form of today's agents, came from here, and Figure 1 of the paper compares four approaches side by side (standard prompting, chain-of-thought, action-only and ReAct), which makes it easy to understand.

Next, read Toolformer paper for 'how a model learns to use tools by itself', Reflexion paper for 'how to learn from failure through text', and CodeAct paper for 'how to express actions as code'; together these cover the three branches of the foundational work. For the overall landscape, see the survey on LLM-based autonomous agents paper.

If you are interested in evaluation, look at WebArena paper and SWE-bench paper. For recent trends, reading 'When Tools Lie' paper, 'From Evidence to Action' paper and 'Judged Useless, Queried Anyway' paper in that order makes the current question, 'how to use tools reliably', clear.