AI TrackerFrontier desk · AEST
Back to news
normalaitech

On the estimation and validity of AI time horizons---a statistical look at the METR plot

arXiv cs.AI · 9 Oct 2026, 03:59 AEST

The study analyzes AI time horizons using METR's 50% time horizon metric, which measures human completion time for tasks solvable by AI. It finds that the difficulty of tasks does not depend linearly on human time, leading to improved time-horizon estimates and diagnostic plots for validity assessment.

Key facts

Read aloud

The study analyzes A I time horizons using METR's fifty percent time horizon metric, which measures human completion time for tasks solvable by A I. It finds that the difficulty of tasks does not depend linearly on human time, leading to improved time-horizon estimates and diagnostic plots for validity assessment. METR's fifty percent time horizon measures the human completion time of software tasks that an A I solves with fifty percent probability, allowing A I capabilities to be expressed in interpretable units. On two hundred twenty eight tasks and twenty six AIs, we recompute the time horizons using splines and item-response theory to relax the assumption that the A I difficulty of a task depends linearly on the log of human time. Our fitted spline can be interpreted as a function that converts human time to A I difficulty; it is nearly flat in a region from two--thirty min but close to linear elsewhere. Overall, we contribute time-horizon point estimates that perform better under a cross-validated suite of proper scoring rules, as well as diagnostic plots for assessing time horizons' construct validity.

Sources in this cluster