MULAN: Multimodal Protein Language Model for Sequence and Structure Encoding

Daria Frolova, Marina A. Pak, Anna Litvin, Ilya Sharov, Dmitry N. Ivankov, Ivan Oseledets

Keywords: Protein language model, protein structure, multimodal model, downstream tasks

TL;DR: New structure-aware protein language model

Abstract:

Most protein language models (PLMs), which produce high-quality protein representations, use only protein sequences during training.
However, the known protein structure is crucial in many protein property prediction tasks, so there is a growing interest in incorporating the knowledge about the protein structure into a PLM. Currently, structure-aware PLMs are trained from scratch or introduce a huge parameter overhead for the structure encoder. In this study, we propose MULAN, a MULtimodal PLM for both sequence and ANgle-based structure encoding. MULAN has a pre-trained sequence encoder and an introduced parameter-efficient Structure Adapter, which are then fused and trained together. According to the evaluation on 9 downstream tasks, MULAN models of various sizes show quality improvement compared to both sequence-only ESM2 and structure-aware SaProt as well as comparable performance to Ankh, ESM3, ProstT5, and other PLMs considered in the study. Importantly, unlike other models, MULAN offers a cheap increase in the structural awareness of the protein representations due to finetuning of existing PLMs instead of training from scratch. We perform a detailed analysis of the proposed model and demonstrate its awareness of the protein structure.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6095