Boost Protein Language Model with Injected Structure Information through Parameter Efficient Fine-tuning

Yuzhe Zhou, Jiayou Zheng, Zixun Zhang, Chun-Mei Feng, Shuguang Cui, Zhen Li

Keywords: Protein Language Model, Parameter-Efficient Fine-Tuning, Structure Information Injecting, ESM2

TL;DR: We introduce a PEFT approach that incorporates structural information into PLM, which could enhance PLM for downstream tasks.

Abstract: At the intersection of computer vision and computational biology, large-scale Protein Language Models (PLMs), particularly the ESM series, have made significant advances in understanding protein structures and functions. However, these models are mainly pre-trained on pure residue sequence, often lack explicit incorporation of structural information, highlighting an opportunity for enhancement. In this paper, we design a parameter-efficient fine-tuning method, SI-Tuning, that injects structural information into PLMs while preserving the original model parameters frozen and optimizing a minimal task-specific vector for input embedding and attention map. This vector, extracted from structural features like dihedral angles and distance maps, introduces a structural bias that enhances the model's performance in downstream tasks. Extensive experiments show that our parameter-efficient fine-tuned ESM-2 650M model outperforms SaProt, a large-scale model pre-trained with protein structural data, in various downstream tasks with a reduction of 40.3% GPU memory and 39.8% time consumption.

Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6327