A standard transformer and attention with linear biases for molecular conformer generation

Viatcheslav Gurev, Timothy Rumbell

TL;DR: A diffusion model with standard transformer for molecular conformer generation that uses attention with linear biases for positional encoding.

Abstract: Sampling low-energy molecular conformations, spatial arrangements of atoms in a molecule, is a critical task for many different calculations performed in the drug discovery and optimization process. Numerous specialized equivariant networks have been designed to generate molecular conformations from 2D molecular graphs. Recently, non-equivariant transformer models have emerged as a viable alternative due to their capability to scale to improve generalization. However, the concern has been that non-equivariant models require a large model size to compensate the lack of equivariant bias. In this paper, we demonstrate that a well-chosen positional encoding effectively addresses these size limitations. A standard transformer model incorporating relative positional encoding for molecular graphs when scaled to 25 million parameters surpasses the current state-of-the-art non-equivariant base model with 64 million parameters on the GEOM-DRUGS benchmark. We implemented relative positional encoding as a negative attention bias that linearly increases with the shortest path distances between graph nodes at varying slopes for different attention heads, similar to ALiBi, a widely adopted relative positional encoding technique in the NLP domain. This architecture has the potential to serve as a foundation for a novel class of generative models for molecular conformations.

Primary Area: Applications->Chemistry, Physics, and Earth Sciences

Keywords: molecular conformers, molecular conformer generation, 3D generation, molecular generative models, equivariant graph networks, diffusion models, transformers

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Submission Number: 7699