Centered Ablation: Why Last-Layer Directional Ablation Under-Fires or Collapses, and a Closed-Form Fix
arXiv cs.CL · 16 Nov 2026, 03:30 AEST
Centered Ablation addresses issues with last-layer directional ablation in language models. The method improves attack success rates and reduces text collapse significantly across multiple models.
Key facts
- Directional ablation removes a hidden state's component along a 'behaviour direction' while a language model runs.
- At the last residual stream, ablation is exactly a logit bias: a fixed vocabulary footprint scaled by a gate, the cosine between the hidden state and the direction.
- Centered Ablation (CA) projects the common mode out of the direction, in closed form.
- Across 8 models, CA raises attack success by 29 and 46 points where the standard recipe under-fires, cuts collapse on Gemma-2-9B from 50% to 1%, and preserves 12 to 40 points more benign answers on high-overlap models.
Read aloud
Centered Ablation addresses issues with last-layer directional ablation in language models. The method improves attack success rates and reduces text collapse significantly across multiple models. Directional ablation removes a hidden state's component along a 'behaviour direction' while a language model runs. At the last residual stream, ablation is exactly a logit bias: a fixed vocabulary footprint scaled by a gate, the cosine between the hidden state and the direction. Centered Ablation CA projects the common mode out of the direction, in closed form. Across eight models, CA raises attack success by twenty nine and forty six points where the standard recipe under-fires, cuts collapse on Gemma-two-9B from fifty percent to one percent, and preserves twelve to forty points more benign answers on high-overlap models.
Sources in this cluster
- Centered Ablation: Why Last-Layer Directional Ablation Under-Fires or Collapses, and a Closed-Form Fix
arXiv cs.CL · 16 Nov 2026, 03:30 AEST