OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport
arXiv cs.CL · 9 Oct 2026, 03:32 AEST
OnTrack is a new streaming monitoring mechanism for LLM agents that alerts users or blocks agents in real-time. It aims to reduce costs and safety issues associated with autonomous agent actions by comparing current steps against successful historical runs.
Key facts
- Agents are deployed in applications from trip planners and stock trading to IT incident triage.
- To overcome this, we propose OnTrack, a streaming monitoring mechanism that compares an agent's steps and dependencies against recorded successful runs to alert users or block the agent in about a millisecond per step.
- Expectation of OnTrack's monitoring capabilities reduces as data access drops, ranging from plan violation detection to identifying loops, stalls, and repeated tool calls.
- Based on the first 8 steps, our method ranks failing trajectories below succeeding ones better than content similarity approaches (+0.057 AUROC).
- With an abort policy, we save about 18% of compute that would be burned on failing runs, where 83% of interrupted runs were actually heading to failure (5 out of 6 aborts were correct).
Read aloud
OnTrack is a new streaming monitoring mechanism for L L M agents that alerts users or blocks agents in real-time. It aims to reduce costs and safety issues associated with autonomous agent actions by comparing current steps against successful historical runs. Agents are deployed in applications from trip planners and stock trading to IT incident triage. To overcome this, we propose OnTrack, a streaming monitoring mechanism that compares an agent's steps and dependencies against recorded successful runs to alert users or block the agent in about a millisecond per step. Expectation of OnTrack's monitoring capabilities reduces as data access drops, ranging from plan violation detection to identifying loops, stalls, and repeated tool calls. Based on the first eight steps, our method ranks failing trajectories below succeeding ones better than content similarity approaches +zero point zero five seven AUROC. With an abort policy, we save about eighteen percent of compute that would be burned on failing runs, where eighty three percent of interrupted runs were actually heading to failure five out of six aborts were correct.
Sources in this cluster
- OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport
arXiv cs.CL · 9 Oct 2026, 03:32 AEST