Automatically Interpreting Millions of Features in Large Language Models

Gonçalo Santos Paulo, Alex Troy Mallen, Caden Juang, Nora Belrose

TL;DR: We build an open-source automated pipeline to generate and evaluate natural language explanations for SAE features using LLMs

Abstract: While the activations of neurons in deep neural networks usually do not have a simple human-understandable interpretation, sparse autoencoders (SAEs) can be used to transform these activations into a higher-dimensional latent space which can be more easily interpretable. However, SAEs can have millions of distinct latents, making it infeasible for humans to manually interpret each one. In this work, we build an open-source automated pipeline to generate and evaluate natural language interpretations for SAE latents using LLMs. We test our framework on SAEs of varying sizes, activation functions, and losses, trained on two different open-weight LLMs. We introduce five new techniques to score the quality of interpretations that are cheaper to run than the previous state of the art. One of these techniques, intervention scoring, evaluates the interpretability of the effects of intervening on a latent, which we find explains latents that are not recalled by existing methods. We propose guidelines for generating better interpretations that remain valid for a broader set of activating contexts, and discuss pitfalls with existing scoring techniques. Our code is available at https://github.com/EleutherAI/delphi.

Lay Summary: Interpretability of large language models remains a open problem. A recent advance in mechanistic interpretability has been the development of sparse autoencoders, a type of explainer model, used to decompose the high dimensional latent space of the base models. In this work we investigate the feasibility of using other large language models to generate explanations for this decompostion and present different methods to evaluate the explanations generated.

Link To Code: https://github.com/EleutherAI/delphi

Primary Area: Social Aspects->Accountability, Transparency, and Interpretability

Keywords: interpretability, language model, sae, features, explanation

Submission Number: 7413