AI TrackerFrontier desk · AEST
Back to news
normalaitech

Holdout Best-of-N: Unbiased Evaluation and Its Cost

arXiv cs.CL · 7 Oct 2026, 03:26 AEST

The study examines the evaluation of Best-of-N selections using a fixed matrix of independent scores. It finds that unbiased evaluation is possible under specific conditions regarding the number of scores used.

Key facts

Read aloud

The study examines the evaluation of Best-of-N selections using a fixed matrix of independent scores. It finds that unbiased evaluation is possible under specific conditions regarding the number of scores used. Reusing the scores that select a Best-of-$N$ winner can overstate its expected reward. A single estimator based only on this matrix is exactly unbiased for expected judge reward under every independent, stable collection of candidate-specific score laws if and only if $J<K$, for every pool size $M\ge N\ge2. For independent Gaussian scores with common variance and fixed $M\ge N\ge2$, the unbiased minimax risk in this regime is of order $σ^two/\sqrt K$, attained by Holdout; allowing bias improves the rate to $σ^two/K.

Sources in this cluster