Finetuning with Sampling: SFT Learns Better Than You Think
arXiv cs.CL · 2 Oct 2026, 03:45 AEST
Introducing new capabilities to frontier models has long been the goal of posttraining, which predominantly employs supervised finetuning (SFT) and reinforcement learning (RL) to this end. Conventional wisdom dictates that RL enables strong generalization on new tasks without losing existing capabilities, while SFT is prone to weak generalization and catastrophic forgetting.
Key facts
- Introducing new capabilities to frontier models has long been the goal of posttraining, which predominantly employs supervised finetuning (SFT) and reinforcement learning (RL) to this end.
- Conventional wisdom dictates that RL enables strong generalization on new tasks without losing existing capabilities, while SFT is prone to weak generalization and catastrophic forgetting.
- At the same time, SFT can learn from off-policy expert data, whereas RL must rely on a model's ability to find successful trajectories with repeated sampling.
- In our work, we seek to leverage the strength of on-policy learning while utilizing the privileged information contained in off-policy data.
- However, rather than modifying the learning objective to accommodate this data, we instead tailor the data distribution to better suit the learner.
Read aloud
Introducing new capabilities to frontier models has long been the goal of posttraining, which predominantly employs supervised finetuning SFT and reinforcement learning RL to this end. Conventional wisdom dictates that RL enables strong generalization on new tasks without losing existing capabilities, while SFT is prone to weak generalization and catastrophic forgetting. Introducing new capabilities to frontier models has long been the goal of posttraining, which predominantly employs supervised finetuning SFT and reinforcement learning RL to this end. Conventional wisdom dictates that RL enables strong generalization on new tasks without losing existing capabilities, while SFT is prone to weak generalization and catastrophic forgetting. At the same time, SFT can learn from off-policy expert data, whereas RL must rely on a model's ability to find successful trajectories with repeated sampling. In our work, we seek to leverage the strength of on-policy learning while utilizing the privileged information contained in off-policy data. However, rather than modifying the learning objective to accommodate this data, we instead tailor the data distribution to better suit the learner.
Sources in this cluster
- Finetuning with Sampling: SFT Learns Better Than You Think
arXiv cs.CL · 2 Oct 2026, 03:45 AEST