Ensembling Sparse Autoencoders

ICLR 2026 Conference Submission14479 Authors

Keywords: interpretability, feature discovery, language models, sparse autoencoders

TL;DR: We show that ensembling sparse autoencoders improves the activation reconstruction while promoting feature diversity, stability, and downstream performance

Abstract: Sparse autoencoders (SAEs) are used to decompose neural network activations into human-interpretable features. Typically, features learned by a single SAE are used for downstream applications. However, it has recently been shown that SAEs trained with different initial weights can learn different features, demonstrating that a single SAE captures only a limited subset of features that can be extracted from the activation space. Motivated by this limitation, we introduce and formalize SAE ensembles. Furthermore, we propose to ensemble multiple SAEs through $\textit{naive bagging}$ and $\textit{boosting}$. In naive bagging, SAEs trained with different weight initializations are ensembled, whereas in boosting SAEs sequentially trained to minimize the residual error are ensembled. Theoretically, naive bagging and boosting are justified as approaches to reduce reconstruction error. Empirically, we evaluate our ensemble approaches with three settings of language models and SAE architectures. Our empirical results demonstrate that, compared to the base SAE and an expanded SAE that matches the number of features in the ensemble, ensembling SAEs can improve the reconstruction of language model activations, diversity of features, and SAE stability. Additionally, on downstream tasks such as concept detection and spurious correlation removal, ensembling SAEs achieve better performance, showing improved practical utility.

Primary Area: interpretability and explainable AI

Submission Number: 14479