Rethinking Neural Thickets Without Weight Perturbations
arXiv cs.AI · 30 Oct 2026, 03:59 AEST
The Neural Thickets hypothesis suggests pretrained language models are in dense neighborhoods of task-specific experts in weight space. New findings indicate that improvements can be achieved through output diversity and selection methods without the need for weight perturbations.
Key facts
- We reproduce the reported gains, but show that they do not require this geometric explanation.
- Perturbing a single decoder block already recovers much of the improvement, while keeping the model entirely frozen and adding a random bias to its output logits matches or exceeds full-weight perturbations on most reasoning tasks.
- We formalize this view as Perturb-Select-Vote (PSV), a general procedure that generates perturbed predictors, selects high-performing members on a small validation set, and aggregates their predictions.
- Together, these results suggest that the gains attributed to nearby weight-space experts can instead be obtained through output diversity, selection, and voting.
Read aloud
The Neural Thickets hypothesis suggests pretrained language models are in dense neighborhoods of task-specific experts in weight space. New findings indicate that improvements can be achieved through output diversity and selection methods without the need for weight perturbations. We reproduce the reported gains, but show that they do not require this geometric explanation. Perturbing a single decoder block already recovers much of the improvement, while keeping the model entirely frozen and adding a random bias to its output logits matches or exceeds full-weight perturbations on most reasoning tasks. We formalize this view as Perturb-Select-Vote PSV, a general procedure that generates perturbed predictors, selects high-performing members on a small validation set, and aggregates their predictions. Together, these results suggest that the gains attributed to nearby weight-space experts can instead be obtained through output diversity, selection, and voting.
Sources in this cluster
- Rethinking Neural Thickets Without Weight Perturbations
arXiv cs.AI · 30 Oct 2026, 03:59 AEST