Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models
arXiv cs.AI · 8 Oct 2026, 03:37 AEST
A new study explores how to predict the performance of coding agents after training based on their base models. The research introduces methods to evaluate base checkpoints without requiring them to initiate the process from scratch.
Key facts
- End-to-end pass@$K$ tests whether successful behavior already appears in a base model's distribution, but it is a poor fit for agentic coding.
- To bridge this gap, we treat successful post-trained agent trajectories as a lookahead signal of base-model potential.
- Across ten pairs of public base and post-trained models, all three screens rank the cohort in close agreement with post-trained SWE-bench Verified pass@$1$.
Read aloud
A new study explores how to predict the performance of coding agents after training based on their base models. The research introduces methods to evaluate base checkpoints without requiring them to initiate the process from scratch. End-to-end pass@$K$ tests whether successful behavior already appears in a base model's distribution, but it is a poor fit for agentic coding. To bridge this gap, we treat successful post-trained agent trajectories as a lookahead signal of base-model potential. Across ten pairs of public base and post-trained models, all three screens rank the cohort in close agreement with post-trained swe bench Verified pass@one dollars$.
Sources in this cluster
- Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models
arXiv cs.AI · 8 Oct 2026, 03:37 AEST