De Novo Drug Design with Joint Transformers

Adam Izdebski, Ewelina Weglarz-Tomczak, Ewa Maria Szczurek, Jakub M. Tomczak

Keywords: generative model, joint model, transformer, molecule generation, drug design

TL;DR: A joint generative model for simultaneous molecule generation and property prediction.

Abstract: \emph{De novo} drug design requires generating novel molecules outside of the training data and simultaneously predicting their target properties, making it a hard task for machine learning models. To address this, we propose Joint Transformer that combines a Transformer decoder, Transformer encoder and a predictor in a joint generative model with shared weights. We show that training the model with a penalized log-likelihood objective results in simultaneously matching state-of-the-art performance in a molecule generation task and successfully predicting molecular properties of newly sampled molecules. Notably, the jointly trained model decreases the prediction error, as compared to a fine-tuned decoder-only Transformer, by $42$%. We propose a probabilistic black box optimization algorithm employing Joint Transformer for the problem of \emph{de novo} drug design, which generates novel molecules with corresponding target properties better than in the data, outperforming other SMILES-based optimization methods.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6491