FDE-Gym: The Last Mile of AI Delivery
arXiv cs.AI · 16 Oct 2026, 03:57 AEST
FDE-Gym introduces a benchmark for evaluating AI agents in real-world customer delivery scenarios. It highlights the challenges of ambiguous requirements and unstated constraints faced by forward-deployed engineers.
Key facts
- We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects.
- FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions.
- In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks.
- Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance.
- These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.
Read aloud
FDE-Gym introduces a benchmark for evaluating A I agents in real-world customer delivery scenarios. It highlights the challenges of ambiguous requirements and unstated constraints faced by forward-deployed engineers. We introduce FDE-Gym, a benchmark of ninety five end-to-end delivery tasks from twenty customer scenarios across eighteen sectors, each adapted by industry experts from real-world delivery projects. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between zero unchanged system and one reference solution, with penalties for stakeholder interactions. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of zero point five six and resolves only fourteen of the ninety five tasks. Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. These results reveal a substantial gap between solving well-specified tasks and delivering effective A I systems for real customers.
Sources in this cluster
- FDE-Gym: The Last Mile of AI Delivery
arXiv cs.AI · 16 Oct 2026, 03:57 AEST