Cache Engineering for Agents
arXiv cs.CL · 21 Oct 2026, 03:32 AEST
Large Language Model (LLM) agents are increasingly managing complex tasks, where key-value cache I/O and storage costs become significant. A new cache engineering scheme allows agents to control their active KV cache without modifying trajectories or re-prefilling context, improving efficiency and performance.
Key facts
- As these trajectories grow, key-value (KV) cache I/O and storage become the dominant cost.
- Experiments show that a tool-response rolling-drop cache policy improves average benchmark scores by 1.7 points and reduces average cost by 27.7%, compared with conventional threshold-triggered summarization.
- With AKV, it achieves up to 1.2x the output throughput and a 1.5x speedup in P95 request latency.
Read aloud
Large Language Model L L M agents are increasingly managing complex tasks, where key-value cache I/O and storage costs become significant. A new cache engineering scheme allows agents to control their active KV cache without modifying trajectories or re-prefilling context, improving efficiency and performance. As these trajectories grow, key-value KV cache I/O and storage become the dominant cost. Experiments show that a tool-response rolling-drop cache policy improves average benchmark scores by one point seven points and reduces average cost by twenty seven point seven percent, compared with conventional threshold-triggered summarization. With AKV, it achieves up to one.2x the output throughput and a one.5x speedup in P95 request latency.
Sources in this cluster
- Cache Engineering for Agents
arXiv cs.CL · 21 Oct 2026, 03:32 AEST