Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models
arXiv cs.CL · 3 Oct 2026, 03:37 AEST
Pivot-SD introduces an efficient offline self-distillation framework for masked diffusion language models. It focuses on training high-impact commitments, or pivots, to enhance performance in complex reasoning tasks.
Key facts
- Masked diffusion language models (dLMs) offer a promising parallel alternative to autoregressive models for complex reasoning.
- We introduce Pivot-SD, an efficient offline self-distillation framework that supervises only these high-impact commitments (pivots).
- Pivot-SD selects pivots using an information-gain metric measuring uncertainty reduction over the remaining masked positions.
- Using only 200 questions and four rollouts each, Pivot-SD improves LLaDA-8B-Instruct over full-sequence SFT and budget-matched diffusion RL baselines across math and code benchmarks.
Read aloud
Pivot-SD introduces an efficient offline self-distillation framework for masked diffusion language models. It focuses on training high-impact commitments, or pivots, to enhance performance in complex reasoning tasks. Masked diffusion language models dLMs offer a promising parallel alternative to autoregressive models for complex reasoning. We introduce Pivot-SD, an efficient offline self-distillation framework that supervises only these high-impact commitments pivots. Pivot-SD selects pivots using an information-gain metric measuring uncertainty reduction over the remaining masked positions. Using only two hundred questions and four rollouts each, Pivot-SD improves LLaDA-8B-Instruct over full-sequence SFT and budget-matched diffusion RL baselines across math and code benchmarks.
Sources in this cluster
- Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models
arXiv cs.CL · 3 Oct 2026, 03:37 AEST