Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy
arXiv cs.AI · 27 Oct 2026, 03:51 AEST
This report evaluates the capabilities of GPT-6-Astra as an agent controlling a robot using EBench. It compares direct control and harness with policy configurations, highlighting performance variations across different task types.
Key facts
- Frontier multimodal large language models (MLLMs) can control robots without robot-specific training, but how their capabilities compare and interact with post-trained vision-language-action (VLA) and world action model (WAM) policies across manipulation demands remains to be thoroughly evaluated.
- Under direct control, the agent achieves competitive overall performance relative to post-trained policies.
- In the harness with policy, invoking a post-trained VLA or WAM combines the complementary strengths of both components, raising overall task success to the level of the strongest compared post-trained policies while substantially reducing token cost.
- However, performance remains below the per-task upper bound of the standalone components due to suboptimal policy invocation, fixed-instruction execution without subgoals, and grasp disruption when the agent resumes control.
Read aloud
This report evaluates the capabilities of GPT-six-Astra as an agent controlling a robot using EBench. It compares direct control and harness with policy configurations, highlighting performance variations across different task types. Frontier multimodal large language models MLLMs can control robots without robot-specific training, but how their capabilities compare and interact with post-trained vision-language-action VLA and world action model WAM policies across manipulation demands remains to be thoroughly evaluated. Under direct control, the agent achieves competitive overall performance relative to post-trained policies. In the harness with policy, invoking a post-trained VLA or WAM combines the complementary strengths of both components, raising overall task success to the level of the strongest compared post-trained policies while substantially reducing token cost. However, performance remains below the per-task upper bound of the standalone components due to suboptimal policy invocation, fixed-instruction execution without subgoals, and grasp disruption when the agent resumes control.
Sources in this cluster
- Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy
arXiv cs.AI · 27 Oct 2026, 03:51 AEST