Holdout Best-of-N: Unbiased Evaluation and Its Cost
arXiv cs.CL · 7 Oct 2026, 03:26 AEST
The study examines the evaluation of Best-of-N selections using a fixed matrix of independent scores. It finds that unbiased evaluation is possible under specific conditions regarding the number of scores used.
Key facts
- Reusing the scores that select a Best-of-$N$ winner can overstate its expected reward.
- A single estimator based only on this matrix is exactly unbiased for expected judge reward under every independent, stable collection of candidate-specific score laws if and only if $J<K$, for every pool size $M\ge N\ge2.
- For independent Gaussian scores with common variance and fixed $M\ge N\ge2$, the unbiased minimax risk in this regime is of order $σ^2/\sqrt K$, attained by Holdout; allowing bias improves the rate to $σ^2/K.
Read aloud
The study examines the evaluation of Best-of-N selections using a fixed matrix of independent scores. It finds that unbiased evaluation is possible under specific conditions regarding the number of scores used. Reusing the scores that select a Best-of-$N$ winner can overstate its expected reward. A single estimator based only on this matrix is exactly unbiased for expected judge reward under every independent, stable collection of candidate-specific score laws if and only if $J<K$, for every pool size $M\ge N\ge2. For independent Gaussian scores with common variance and fixed $M\ge N\ge2$, the unbiased minimax risk in this regime is of order $σ^two/\sqrt K$, attained by Holdout; allowing bias improves the rate to $σ^two/K.
Sources in this cluster
- Holdout Best-of-N: Unbiased Evaluation and Its Cost
arXiv cs.CL · 7 Oct 2026, 03:26 AEST