AI TrackerFrontier desk · AEST
Back to news
normalaitech

Rethinking Neural Thickets Without Weight Perturbations

arXiv cs.AI · 30 Oct 2026, 03:59 AEST

The Neural Thickets hypothesis suggests pretrained language models are in dense neighborhoods of task-specific experts in weight space. New findings indicate that improvements can be achieved through output diversity and selection methods without the need for weight perturbations.

Key facts

Read aloud

The Neural Thickets hypothesis suggests pretrained language models are in dense neighborhoods of task-specific experts in weight space. New findings indicate that improvements can be achieved through output diversity and selection methods without the need for weight perturbations. We reproduce the reported gains, but show that they do not require this geometric explanation. Perturbing a single decoder block already recovers much of the improvement, while keeping the model entirely frozen and adding a random bias to its output logits matches or exceeds full-weight perturbations on most reasoning tasks. We formalize this view as Perturb-Select-Vote PSV, a general procedure that generates perturbed predictors, selects high-performing members on a small validation set, and aggregates their predictions. Together, these results suggest that the gains attributed to nearby weight-space experts can instead be obtained through output diversity, selection, and voting.

Sources in this cluster