AutoScaffold: AI-Guided Scaffolding for Reinforcement Learning of Language Models
arXiv cs.CL · 11 Nov 2026, 04:58 AEST
AutoScaffold is a new AI framework that enhances reinforcement learning by creating adaptive scaffolds for language model training. It allows for minimal human intervention while improving performance across various domains.
Key facts
- Reinforcement learning with verifiable rewards (RLVR) produces rich training experience, yet outcome rewards reduce each attempt to a single number and leave most of this information unused.
- We introduce AutoScaffold, a unified AI-for-AI framework that turns a student's training experience into adaptive scaffolds, with minimal domain-specific information and no human intervention during training.
- When evaluated without scaffolds, students trained with AutoScaffold outperform those trained with the corresponding RL baselines in all four domains, with gains of 7.2 points in ALFWorld success and 4.0 points in average QA exact match.
- On GPU kernel generation, AutoScaffold increases the fraction of tasks with a correct kernel that runs faster than the PyTorch reference by up to 5.0 points.
- Analyses show that the teacher identifies a different failure in each task class and shifts its scaffolds from step-by-step workflows to targeted fixes as the student improves.
Read aloud
AutoScaffold is a new A I framework that enhances reinforcement learning by creating adaptive scaffolds for language model training. It allows for minimal human intervention while improving performance across various domains. Reinforcement learning with verifiable rewards RLVR produces rich training experience, yet outcome rewards reduce each attempt to a single number and leave most of this information unused. We introduce AutoScaffold, a unified A I-for-A I framework that turns a student's training experience into adaptive scaffolds, with minimal domain-specific information and no human intervention during training. When evaluated without scaffolds, students trained with AutoScaffold outperform those trained with the corresponding RL baselines in all four domains, with gains of seven point two points in ALFWorld success and four point zero points in average QA exact match. On G P U kernel generation, AutoScaffold increases the fraction of tasks with a correct kernel that runs faster than the PyTorch reference by up to five point zero points. Analyses show that the teacher identifies a different failure in each task class and shifts its scaffolds from step-by-step workflows to targeted fixes as the student improves.
Sources in this cluster
- AutoScaffold: AI-Guided Scaffolding for Reinforcement Learning of Language Models
arXiv cs.CL · 11 Nov 2026, 04:58 AEST