Beyond SAEs: Consistent Concept Discovery with Sparse Coding
arXiv cs.AI · 17 Oct 2026, 03:25 AEST
A new method called Consistent Sparse Dictionary Learning (ConSDL) improves the reliability of sparse autoencoders in concept discovery. It achieves significant gains in dictionary and coefficient consistency across various settings.
Key facts
- Sparse autoencoders (SAEs) explain foundation model representations by decomposing activations into sparse combinations of learned dictionary atoms.
- However, the learned dictionaries can vary across training runs, limiting the reliability and reproducibility of SAE-based analyses.
- Across 16 synthetic settings, ConSDL achieves near-perfect dictionary and coefficient recovery.
- These gains translate to state-of-the-art cross-seed consistency across four vision and language settings, where ConSDL improves dictionary and coefficient consistency by 17.6% and 35.9%, respectively, over the average of six SAE baselines.
Read aloud
A new method called Consistent Sparse Dictionary Learning ConSDL improves the reliability of sparse autoencoders in concept discovery. It achieves significant gains in dictionary and coefficient consistency across various settings. Sparse autoencoders SAEs explain foundation model representations by decomposing activations into sparse combinations of learned dictionary atoms. However, the learned dictionaries can vary across training runs, limiting the reliability and reproducibility of SAE-based analyses. Across sixteen synthetic settings, ConSDL achieves near-perfect dictionary and coefficient recovery. These gains translate to state-of-the-art cross-seed consistency across four vision and language settings, where ConSDL improves dictionary and coefficient consistency by seventeen point six percent and thirty five point nine percent, respectively, over the average of six SAE baselines.
Sources in this cluster
- Beyond SAEs: Consistent Concept Discovery with Sparse Coding
arXiv cs.AI · 17 Oct 2026, 03:25 AEST