On the estimation and validity of AI time horizons---a statistical look at the METR plot
arXiv cs.AI · 9 Oct 2026, 03:59 AEST
The study analyzes AI time horizons using METR's 50% time horizon metric, which measures human completion time for tasks solvable by AI. It finds that the difficulty of tasks does not depend linearly on human time, leading to improved time-horizon estimates and diagnostic plots for validity assessment.
Key facts
- METR's 50% time horizon measures the human completion time of software tasks that an AI solves with 50% probability, allowing AI capabilities to be expressed in interpretable units.
- On 228 tasks and 26 AIs, we recompute the time horizons using splines and item-response theory to relax the assumption that the AI difficulty of a task depends linearly on the log of human time.
- Our fitted spline can be interpreted as a function that converts human time to AI difficulty; it is nearly flat in a region from 2--30 min but close to linear elsewhere.
- Overall, we contribute time-horizon point estimates that perform better under a cross-validated suite of proper scoring rules, as well as diagnostic plots for assessing time horizons' construct validity.
Read aloud
The study analyzes A I time horizons using METR's fifty percent time horizon metric, which measures human completion time for tasks solvable by A I. It finds that the difficulty of tasks does not depend linearly on human time, leading to improved time-horizon estimates and diagnostic plots for validity assessment. METR's fifty percent time horizon measures the human completion time of software tasks that an A I solves with fifty percent probability, allowing A I capabilities to be expressed in interpretable units. On two hundred twenty eight tasks and twenty six AIs, we recompute the time horizons using splines and item-response theory to relax the assumption that the A I difficulty of a task depends linearly on the log of human time. Our fitted spline can be interpreted as a function that converts human time to A I difficulty; it is nearly flat in a region from two--thirty min but close to linear elsewhere. Overall, we contribute time-horizon point estimates that perform better under a cross-validated suite of proper scoring rules, as well as diagnostic plots for assessing time horizons' construct validity.
Sources in this cluster
- On the estimation and validity of AI time horizons---a statistical look at the METR plot
arXiv cs.AI · 9 Oct 2026, 03:59 AEST