Multi-Modal and Multi-Task Transformer for Small Molecule Drug Discovery

Sai Krishna Sirumalla, David Stephen Farina Jr, Zhuoran Qiao, Daniele Alessandro Di Cesare, Felipe Costas Farias, Michael Bernard O’Connor, Peter John Bygrave, Feizhi Ding, Thomas Dresselhaus, Marcelo Gomes Pereira de Lacerda, Jason Matthew Swails, Daniel Miles, Matthew Welborn, Fred Manby, Thomas Miller

Keywords: Drug discovery, Transformers, Multi-modal, Multi-task, Molecular property prediction

Abstract: We introduce a 1B-parameter transformer model pre-trained from scratch on 2.25T tokens from a massive mixture of datasets centered around drug discovery. These datasets are heterogeneous, coming from dozens of sources and covering 15 data modalities. We demonstrate the model’s capability on various molecular assay prediction tasks, including public benchmarks and internally generated holdouts from real-world drug discovery programs. Following parameter-efficient fine-tuning, the multi-modal transformer excels at multi-task predictions compared to strong molecular property prediction baselines including XGBoost and Chemprop.

Submission Number: 110