What Do RTL Repair Benchmarks Actually Measure?
arXiv cs.AI · 21 Oct 2026, 03:45 AEST
RTL repair benchmarks are designed to assess a model's ability to fix broken RTL based on a specification. The study identifies measurement confounds and proposes evaluation controls to improve the reliability of these benchmarks.
Key facts
- RTL repair benchmarks should measure whether a model can fix broken RTL given a specification.
- We construct and hash-bind RTLRepairDataset, 5,784 (broken, fixed) training pairs; RTLRepairBench, 459 held-out cases with 374 syntactic and 85 behavioral mutations; and an evaluation harness supporting simulation and formal equivalence.
- We then use this infrastructure to identify two measurement confounds.
- Thus, a conventional repair score need not isolate whether the model used the faulty RTL.
- Together, the resources and audit provide both a benchmark for behavioral repair and a protocol for diagnosing what contributes to its measured score.
Read aloud
RTL repair benchmarks are designed to assess a model's ability to fix broken RTL based on a specification. The study identifies measurement confounds and proposes evaluation controls to improve the reliability of these benchmarks. RTL repair benchmarks should measure whether a model can fix broken RTL given a specification. We construct and hash-bind RTLRepairDataset, five thousand seven hundred eighty four broken, fixed training pairs; RTLRepairBench, four hundred fifty nine held-out cases with three hundred seventy four syntactic and eighty five behavioral mutations; and an evaluation harness supporting simulation and formal equivalence. We then use this infrastructure to identify two measurement confounds. Thus, a conventional repair score need not isolate whether the model used the faulty RTL. Together, the resources and audit provide both a benchmark for behavioral repair and a protocol for diagnosing what contributes to its measured score.
Sources in this cluster
- What Do RTL Repair Benchmarks Actually Measure?
arXiv cs.AI · 21 Oct 2026, 03:45 AEST