Paper review · October 23, 2026
Cache Engineering for Agents
The authors let agents delete and reposition the KV cache directly and co-designed a dedicated serving engine, AKV, cutting cost by 27.7% and raising scores by 1.7 points compared with summarization.
Problem
As agent trajectories grow longer, the cost of storing and reading the KV cache rises. They can also exceed the context length limit. Existing context engineering (truncating or summarizing history) modifies earlier parts, which invalidates the cache after that point and requires re-prefilling. Truncating heavily to reduce prefill cost lowers accuracy, creating a dilemma.
Method
The authors propose "cache engineering," which leaves the text trajectory unchanged and manipulates only the active KV cache. In addition to the existing append, they define drop (excluding selected cache entries from later attention) and repos (reassigning the position indices of the remaining keys via RoPE rotation) operations. Even for the same token sequence, a different order of operations leads to a different cache state, creating a "history dependence" problem. To solve this, they designed the serving engine AKV. Externally, AKV provides a stateless interface compatible with existing APIs; internally, it uses an extended radix tree with drop nodes, drop-aware eviction, and segment-batch attention that restores the cache in a single prefill. The main policy is tool-response rolling drop, which discards old tool-response cache entries in sequence.
Results
- Across 3 LLMs and 4 agent benchmarks, the tool-response rolling drop policy reduced average cost by 27.7% and raised the average score by 1.7 points compared with threshold-based summarization.
- On the AKV server, output throughput improved 1.18× and P95 request latency improved 1.51×.
- A policy found through recursive self-improvement (RSI) reduced cost by an additional 7.1% on held-out tasks compared with an expert-designed policy.
- In a motivating experiment (GPT-OSS-120B, 100 BCP tasks), truncation-based context engineering cost 1.85× as much as the full context. Summarization with a 48k threshold reduced cost by about 7% but lowered accuracy by 5 percentage points.
Limitations
Because drop creates cache states that did not exist during original training, its effect may vary by model and policy. Validation covers 3 LLMs and 4 benchmarks. The serving gains require the AKV engine, so they are hard to apply directly to commercial APIs where the cache cannot be controlled. The approach assumes RoPE-based repositioning.
Why it matters
The paper points out that most of the cost of running long-horizon agents comes from reading the KV cache. It opens a new design option of managing context at the cache level instead of editing it as text, and is directly relevant to cutting the cost of agent serving infrastructure.
Key terms
- KV cache
- Memory that stores the Key and Value values of already processed tokens and reuses them when generating the next token. The longer the context, the higher the cost of storing and reading it.
- Prefill
- The stage that processes the entire input prompt at once to build the KV cache. If an earlier part changes, everything after it must be prefilled again.
- Context engineering
- Techniques that truncate or summarize an agent's conversation history to manage model input length and cost.
- RoPE
- A method that adds position information by rotating Key and Query vectors according to their position. Reapplying the rotation can change the positions of cached keys.
Original
Authors: Tianyu Fu, Ruoxi Wang, Zihan Min, Zhongqi Ren, Zhihao Li, Tengxuan Liu, Shengxu Mu, Hanling Zhang, et al. · Posted 2026-10-20 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.