When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
arXiv cs.CL · 7 Oct 2026, 03:25 AEST
A study explores the concept of spurious forgetting in language models during finetuning. It reveals that knowledge often remains recoverable and that forgetting can be non-catastrophic under certain conditions.
Key facts
- Knowledge that a language model appears to forget during finetuning often remains stored and can be recovered, a phenomenon called spurious forgetting.
- Finetuning on new facts can even produce forgetting that undoes itself: recall of the old facts collapses, recovers as training continues on new facts alone, and only then erodes for good.
- A minimal associative memory reproduces these dynamics with three ingredients: keys with shared structure, concentrated new values, and normalization in the network.
Read aloud
A study explores the concept of spurious forgetting in language models during finetuning. It reveals that knowledge often remains recoverable and that forgetting can be non-catastrophic under certain conditions. Knowledge that a language model appears to forget during finetuning often remains stored and can be recovered, a phenomenon called spurious forgetting. Finetuning on new facts can even produce forgetting that undoes itself: recall of the old facts collapses, recovers as training continues on new facts alone, and only then erodes for good. A minimal associative memory reproduces these dynamics with three ingredients: keys with shared structure, concentrated new values, and normalization in the network.
Sources in this cluster
- When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
arXiv cs.CL · 7 Oct 2026, 03:25 AEST