Circuits, Features, and Heuristics in Molecular Transformers

ICLR 2026 Conference Submission24132 Authors

Keywords: Computational Chemistry, Drug Discovery, Mechanistic Interpretability, Molecular Generation, Transformers, Sparse Autoencoders

Abstract: Transformers generate valid and diverse chemical structures, but little is known about the mechanisms that enable these models to capture the rules of molecular representation. We present a mechanistic analysis of autoregressive transformers trained on drug-like small molecules to reveal the computational structure underlying their capabilities across multiple levels of abstraction. We identify computational patterns consistent with low-level syntactic parsing and more abstract chemical validity constraints. Using sparse autoencoders (SAEs), we extract feature dictionaries associated with chemically relevant activation patterns. We validate our findings on downstream tasks and find that mechanistic insights can translate to predictive performance in various practical settings.

Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)

Submission Number: 24132