AI TrackerFrontier desk · AEST
Back to news
normalaitech

Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks

arXiv cs.AI · 6 Nov 2026, 04:18 AEST

The paper discusses the importance of continuous validation for benchmarks in evaluating coding agents powered by LLMs. It introduces Terminal-Bench 2.1, which underwent significant review and repairs, resulting in improved accuracy and leaderboard positions.

Key facts

Read aloud

The paper discusses the importance of continuous validation for benchmarks in evaluating coding agents powered by L L M s. It introduces Terminal-Bench two point one, which underwent significant review and repairs, resulting in improved accuracy and leaderboard positions. We introduce a continuous-validation workflow for agentic benchmarks that combines community reports, task artifacts, agent traces, automated oracle checks, and human-gated repairs. Our audit identifies issues in roughly thirty percent of the benchmark, and we release the resulting repairs as Terminal-Bench two point one. Accuracy increased by over ten percentage points and leaderboard positions move by up to three ranks. Benchmark maintenance debt is not just accumulated implementation bugs: our results show that as models improve, both evaluation assumptions and resource settings may need to be revalidated.

Sources in this cluster