Fine-Tuning of Conditional Transformers Improves the Generation of Functionally Characterized Proteins

Marco Nicolini, Dario Malchiodi, Alberto Cabri, Emanuele Cavalleri, Marco Mesiti, Alberto Paccanaro, Peter N. Robinson, Justin T. Reese, Elena Casiraghi, Giorgio Valentini

Abstract: Conditional transformers improve the generative capabilities of large language models (LLMs) by processing specific control tags able to drive the generation of texts characterized by specific features. Recently, a similar approach has been applied to the generation of functionally characterized proteins by adding specific tags to the protein sequence to qualify their functions (e.g., Gene Ontology terms) or other characteristics (e.g., their family or the species which they belong to). In this work, we show that fine tuning conditional transformers, pre-trained on large corpora of proteins, on specific protein families can significantly enhance the prediction accuracy of the pre-trained models and can also generate new potentially functional proteins that could enlarge the protein space explored by the natural evolution. We obtained encouraging results on the phage lysozyme family of proteins, achieving statistically significant better prediction results than the original pre-traine