Interpretability as Compression: Reconsidering SAE Explanations of Neural Activations
Keywords: mechanistic interpretability, sparse autoencoders, compression, feature splitting, intermediate representations
TL;DR: Optimising for low description length instead of sparsity in SAEs avoids undesirable feature splitting, inspires novel SAE architectures and results in more interpretable features
Abstract: Sparse Autoencoders (SAEs) have emerged as a useful tool for interpreting the internal representations of neural networks. However, naively optimising SAEs for reconstruction loss and sparsity results in a preference for SAEs that are extremely wide and sparse.
We present an information-theoretic framework for interpreting SAEs as lossy compression algorithms for communicating explanations of neural activations. We appeal to the Minimal Description Length (MDL) principle to motivate explanations of activations which are both accurate and concise. We further argue that interpretable SAEs require an additional property, “independent additivity”: features should be able to be understood separately. We demonstrate an example of applying our MDL-inspired framework by training SAEs on MNIST handwritten digits and find that SAE features representing significant line segments are optimal, as opposed to SAEs with features for memorised digits from the dataset or small digit fragments.
We argue that using MDL rather than sparsity may avoid potential pitfalls with naively maximising sparsity such as undesirable feature splitting and that this framework naturally suggests new hierarchical SAE architectures which provide more concise explanations.
Track: Main track
Submitted Paper: No
Published Paper: No
Submission Number: 65
Loading