3D Molecular Pretraining via Localized Geometric Generation
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: molecular repsentation; self-supervised learning
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: Self-supervised learning on 3D molecular structures has gained prominence in AI-driven drug discovery due to the high cost of annotating biochemical data.
However, few have studied the selection of proper modeling semantic units within 3D molecular data, which is critical for an expressive pre-trained model as recognized in natural language processing and computer vision.
In this study, we introduce \textbf{L}ocalized G\textbf{e}ometric \textbf{G}enerati\textbf{o}n (LEGO), a novel approach that treats tetrahedrons within 3D molecular structures as fundamental building blocks , leveraging their simplicity in three-dimension and their prevalence in molecular structural patterns such as carbon skeletons and functional groups.
Inspired by masked language/image modeling, LEGO perturbs a portion of tetrahedrons and learns to reconstruct them in pretraining.
The reconstruction of the noised local structures can be divided into a two-step process, namely spatial orientation prediction and internal arrangement generation.
First, we predict the global orientation of the noised local structure within the whole molecule, equipping the model with positional information for these foundational components.
Then, we geometrically reconstruct the internal arrangements of the noised local structures revealing their functional semantics.
To address the atom-bond inconsistency problem in previous denoising methods and utilize the prior of chemical bonds, we propose to model the graph as a set of nodes and edges and explicitly generate the edges during pre-training.
In this way, LEGO exploits the advantages of encoding structural geometry features as well as leveraging the expressiveness of self-supervised learning.
Extensive experiments on molecular quantum and biochemical property prediction tasks demonstrate the effectiveness of our approach.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 9430
Loading