Comparing Protein Language Models Using Remote Homology Detection for Phages
Keywords: Protein Language Model, Machine Learning, Classification, Virus
TL;DR: We compare five protein language models by applying them to the unsolved biological problem of remote phage homology.
Abstract: Background. Protein language models (pLMs) are machine learning models that
learn high-dimensional representations of protein sequences. These models have
utility in biological settings, for instance pLMs can convert between protein sequence and structure (Heinzinger et al., 2023), determine evolutionary relationships between organisms (Bordin et al., 2023), and design protein sequences with
desired functions (Madani et al., 2023). Transfer learning with previously trained
pLMs offers a powerful, minimal resource strategy for performing diverse largescale classification and prediction tasks. However, as pLMs proliferate in the
research community with differences in training objectives, model structure(s)
and training datasets, it is daunting for a less-experienced end user to decide
which pLM to use for biological experiments and discovery. Consequently, it
is essential to compare pLMs to determine their strengths and limitations. In
particular, such explorations are necessary in use-cases relevant to biological researchers. Therefore, we present a comparison of pre-trained pLMs in a difficult remote homology detection task for phage proteins described previously in
Flamholz et al. (2024), Large language models improve annotation of prokaryotic
viral proteins. We also make available our code and notebooks to facilitate other
research scientists to use such models. Results. Variations in model training resulted in
significantly different performance in our biological task. We present an analysis that compares five recently published pLMs : (1) ProtT5, (2) ProstT5, (3)
TMVec, (4) ESM2, and (5) CARP. We observed that all models were able to capture meaningful structural information in viral proteins. We also determined that
their embeddings could be used to train functional classifiers that, when tested
using the PHROG and EFAM databases of phage proteins, captured meaningful
biological information. However, the performances across the different models
were noticeably different. Models trained on larger, more diverse databases of
genomic sequences such as Big Fantastic Database (BFD) performed better overall. Models with the Transformer architecture performed better than those with
the convolutional neural network (CNN) architectures. Conclusion. The utility
of pLMs in areas of biological research is clear as we demonstrate such models
are useful for remote homology detection in phage genomes, an area of active
interest in metagenomics and environmental biology. Our study highlights how
biological scientists can choose pLMs to incorporate into their experiments and
analyses. Overall, while some models clearly performed better, on the whole, all
pLMs achieved high scores for prediction. For end-users, the implication is that
many pLM models are useful, but testing and domain knowledge may improve
results when addressing specific biological questions and developing specialized
model training paradigms.
Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 12075
Loading