Advancing the Boundary of Pre-Trained Models for Drug Discovery: Interpretable Fine-Tuning Empowered by Molecular Physicochemical Properties

Xiaoqing Lian, Jie Zhu, Tianxu Lv, Xiaoyan Hong, Longzhen Ding, Wei Chu, Jianming Ni, Xiang Pan

Abstract: In the field of drug discovery, a proliferation of pre-trained models has surfaced, exhibiting exceptional performance across a variety of tasks. However, the extensive size of these models, coupled with the limited interpretative capabilities of current fine-tuning methods, impedes the integration of pre-trained models into the drug discovery process. This paper pushes the boundaries of pre-trained models in drug discovery by designing a novel fine-tuning paradigm known as the Head Feature Parallel Adapter (HFPA), which is highly interpretable, high-performing, and has fewer parameters than other widely used methods. Specifically, this approach enables the model to consider diverse information across representation subspaces concurrently by strategically using Adapters, which can operate directly within the model's feature space. Our tactic freezes the backbone model and forces various small-size Adapters' corresponding subspaces to focus on exploring different atomic and chemical bond knowledge, thus maintaining a small number of trainable parameters and enhancing the interpretability of the model. Moreover, we furnish a comprehensive interpretability analysis, imparting valuable insights into the chemical area. HFPA outperforms over seven physiology and toxicity tasks and achieves state-of-the-art results in three physical chemistry tasks. We also test ten additional molecular datasets, demonstrating the robustness and broad applicability of HFPA.