Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
arXiv cs.AI · 6 Nov 2026, 04:18 AEST
The paper discusses the importance of continuous validation for benchmarks in evaluating coding agents powered by LLMs. It introduces Terminal-Bench 2.1, which underwent significant review and repairs, resulting in improved accuracy and leaderboard positions.
Key facts
- We introduce a continuous-validation workflow for agentic benchmarks that combines community reports, task artifacts, agent traces, automated oracle checks, and human-gated repairs.
- Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1.
- Accuracy increased by over 10 percentage points and leaderboard positions move by up to three ranks.
- Benchmark maintenance debt is not just accumulated implementation bugs: our results show that as models improve, both evaluation assumptions and resource settings may need to be revalidated.
Read aloud
The paper discusses the importance of continuous validation for benchmarks in evaluating coding agents powered by L L M s. It introduces Terminal-Bench two point one, which underwent significant review and repairs, resulting in improved accuracy and leaderboard positions. We introduce a continuous-validation workflow for agentic benchmarks that combines community reports, task artifacts, agent traces, automated oracle checks, and human-gated repairs. Our audit identifies issues in roughly thirty percent of the benchmark, and we release the resulting repairs as Terminal-Bench two point one. Accuracy increased by over ten percentage points and leaderboard positions move by up to three ranks. Benchmark maintenance debt is not just accumulated implementation bugs: our results show that as models improve, both evaluation assumptions and resource settings may need to be revalidated.
Sources in this cluster
- Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
arXiv cs.AI · 6 Nov 2026, 04:18 AEST