AI TrackerFrontier desk · AEST
Back to news
normalaitech

Simulated Control Evals

arXiv cs.AI · 23 Oct 2026, 03:26 AEST

Simulated control evaluations (SCEs) use a language model to simulate tool responses, aiming to improve the effectiveness of AI safeguards. The results indicate that SCEs can predict safety more accurately than traditional control settings like BashArena.

Key facts

Read aloud

Simulated control evaluations SCEs use a language model to simulate tool responses, aiming to improve the effectiveness of A I safeguards. The results indicate that SCEs can predict safety more accurately than traditional control settings like BashArena. Control evaluations measure whether safeguards can prevent an A I model from causing undesired outcomes even when it is intentionally trying to do so. We introduce simulated control evaluations SCEs, which use a language model to simulate tool responses in place of an environment. For independently sampled evaluation and deployment task sets, SCEs predict safety with a root mean squared error of three point six percentage points across five monitors at a five percent audit budget, compared with ten point one for BashArena. SCEs closely reproduce distributions of attack strategies, attacker reasoning, and monitor reasoning, with over tenfold lower KL divergence from execution than BashArena. These results suggest that deployment-specific SCEs can be more predictive than control evaluations executed in separate control settings.

Sources in this cluster