Your Transformer Is Secretly Elastic
arXiv cs.AI · 15 Oct 2026, 03:58 AEST
A new method allows pre-trained transformers to adapt dynamically to varying computational budgets. This approach enhances efficiency without the need for retraining, achieving significant performance improvements.
Key facts
- We introduce a retraining-free method that elastifies pre-trained transformers into a single model that adapts on-the-fly across a continuum of computational budgets.
- Our key contribution is a differentiable relaxation of the functional unit-ranking problem: unlike prior work, which optimizes a single importance weight per MLP block or attention head, we rank every individual MLP neuron and attention head jointly.
- Across 27 vision and text transformer encoders ranging from 22M to 8B parameters, our method yields smooth degradation up to 60% sparsity and outperforms one-shot baselines under deep pruning.
- For example, pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters while reducing ImageNet-1k linear-probing accuracy by only 0.8%, without the need for weight correction or retraining.
- At 60% sparsity, a pruned AugReg ViT-B/16 model achieves up to $1.80\times$ higher throughput and $1.26\times$ lower single-query latency on an A100 GPU.
Read aloud
A new method allows pre-trained transformers to adapt dynamically to varying computational budgets. This approach enhances efficiency without the need for retraining, achieving significant performance improvements. We introduce a retraining-free method that elastifies pre-trained transformers into a single model that adapts on-the-fly across a continuum of computational budgets. Our key contribution is a differentiable relaxation of the functional unit-ranking problem: unlike prior work, which optimizes a single importance weight per MLP block or attention head, we rank every individual MLP neuron and attention head jointly. Across twenty seven vision and text transformer encoders ranging from 22M to 8B parameters, our method yields smooth degradation up to sixty percent sparsity and outperforms one-shot baselines under deep pruning. For example, pruning DINOv3 ViT-H+/sixteen to fifty percent sparsity removes 427M parameters while reducing ImageNet-1k linear-probing accuracy by only zero point eight percent, without the need for weight correction or retraining. At sixty percent sparsity, a pruned AugReg ViT-B/sixteen model achieves up to one dollars and eighty cents\times$ higher throughput and one dollars and twenty six cents\times$ lower single-query latency on an A100 G P U.
Sources in this cluster
- Your Transformer Is Secretly Elastic
arXiv cs.AI · 15 Oct 2026, 03:58 AEST