Paper review · October 30, 2026
TAO: tool-aware joint scheduling for serving LLM agents
TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving
By jointly accounting for when tool executions return, time-decayed KV cache value and token volume, and by optimizing session selection and placement together with dynamic programming, the system reduced latency and raised throughput in agent serving.
Problem
Agent sessions alternate between LLM inference on GPU clusters and external tool execution (compilers, search and so on) on distributed CPUs. Existing serving systems do not consider when tools will finish when valuing a session, which wastes resources. They also decide admission and node placement with per-session heuristic rules, leading to short-sighted choices among sessions competing for the same node capacity.
Method
A session that is calling a tool releases its GPU resources and enters a waiting pool. Each session–node pair is assigned a triple: KV cache credit decayed with a 10-second half-life, a binary readiness flag indicating whether the tool will return within 1 second, and the token count. Readiness is estimated from tool progress hints, parsing of sleep commands and past tool execution times. A multiple knapsack problem that maximizes this value lexicographically is solved exactly with capacity-state dynamic programming, deciding waiting-session selection and node placement at the same time. Planning runs on an asynchronous worker and is validated against the live state just before it is applied.
Results
- Across 12 model, workload and concurrency settings, compared with the best baseline for each metric (Dynamo, vLLM RR/CA, ThunderAgent), it reduced latency by up to 26.8% and mean TTFT by up to 36.3%, and raised decode throughput by up to 29.5%.
- The largest improvement came in the DeepSeek-V4 Flash (2 nodes with 4 H100s each), SWE-Lite, 64-worker setting (latency 185.4 s vs. 253.2 s for the best baseline).
- On Terminal-Bench, TTFT improved by 16–22%, but decode throughput was 8.8% and 6.2% lower than the baseline at 16 and 64 workers, respectively.
- Time spent preparing and solving plans was under 0.12% of total runtime in the 4-replica experiments.
Limitations
It does not win in every setting. With Qwen3-32B on SWE-verified at 128 workers, latency was 2.5% worse, and in some settings decode throughput was below the baseline. The experiments are limited to small clusters of about 8 H100s and SWE- and Terminal-type coding agent workloads. Because the dynamic programming discretizes capacity, rounding can exclude placements that are actually feasible. Parameters such as the half-life and time window were fixed.
Why it matters
In services with frequent tool calls, such as coding agents, GPUs easily sit idle waiting for tools or cache is wasted. This paper shows that a scheduler that uses tool execution timing as a signal can improve latency and throughput on the same hardware.
Key terms
- CGDC(Clustered GPUs with Distributed CPUs)
- The authors' name for an agent serving architecture in which inference runs on GPU clusters and tool execution runs on external distributed CPUs.
- KV cache reuse
- An approach that reuses the KV cache computed in a previous turn for the next request, skipping prefill computation. It only helps if the session is placed on the same node.
- TTFT(Time To First Token)
- The time from a request until the first token appears; a standard metric for perceived response speed.
- Multiple knapsack problem
- A combinatorial optimization problem of dividing items among several bags of fixed capacity so that the total value is maximized. Here, nodes are the bags and sessions are the items.
Original
Authors: Shuhuai Li, Shengdong Ye, Dongdong Ge, Yinyu Ye, Jianghao Lin · Posted 2026-10-27 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.