Masked inverse folding with sequence transfer for protein representation learning

Kevin K Yang, Hugh Yeh, Niccolò Zanichelli

Abstract:

Self-supervised pretraining on protein sequences has led to state-of-the art performance on protein function and fitness prediction.
However, sequence-only methods ignore the rich information contained in experimental and predicted protein structures.
Meanwhile, inverse folding methods reconstruct a protein's amino-acid sequence given its structure, but do not take advantage of sequences that do not have known structures.
In this study, we train a masked inverse folding protein language model parameterized as a structured graph neural network.
We then show that using the outputs from a pretrained sequence-only protein masked language model as input to the inverse folding model further improves pretraining perplexity.
We evaluate both of these models on downstream protein engineering tasks and analyze the effect of using information from experimental or predicted structures on performance.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Machine Learning for Sciences (eg biology, physics, health sciences, social sciences, climate/sustainability )