AI TrackerFrontier desk · AEST
Back to news
normalaitech

FDE-Gym: The Last Mile of AI Delivery

arXiv cs.AI · 16 Oct 2026, 03:57 AEST

FDE-Gym introduces a benchmark for evaluating AI agents in real-world customer delivery scenarios. It highlights the challenges of ambiguous requirements and unstated constraints faced by forward-deployed engineers.

Key facts

Read aloud

FDE-Gym introduces a benchmark for evaluating A I agents in real-world customer delivery scenarios. It highlights the challenges of ambiguous requirements and unstated constraints faced by forward-deployed engineers. We introduce FDE-Gym, a benchmark of ninety five end-to-end delivery tasks from twenty customer scenarios across eighteen sectors, each adapted by industry experts from real-world delivery projects. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between zero unchanged system and one reference solution, with penalties for stakeholder interactions. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of zero point five six and resolves only fourteen of the ninety five tasks. Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. These results reveal a substantial gap between solving well-specified tasks and delivering effective A I systems for real customers.

Sources in this cluster

FDE-Gym: The Last Mile of AI Delivery · AI Tracker