SweetBERT: exploring BERT-based models for IUPAC glycan nomenclature modeling

Irene Rubia-Rodríguez, Henrik Nielsen, Garry P. Gippert, Kristian Barrett, Bernard Henrissat, Ole Winther

Track: Machine learning: computational method and/or computational results

Nature Biotechnology: Yes

Keywords: Glybiology, BERT, Transformers, Language Models

TL;DR: This work introduces SweetBERT, a BERT-based model that leverages Transformer architectures to capture the complexity of IUPAC glycan nomenclature by including a pseudo-graph representation of the branched structure of glycans.

Abstract: Glycans are the most abundant biomolecules on Earth, and participate in key processes in all living organisms. The chemical variability and topological complexity of their natural branched structures has been a challenge in computational glycobiology. As a tool for improving predictive models associated with glycobiology, we propose SweetBERT, a BERT-based language model for encoding glycan sequences which includes explicit information about the branching structure of the sequence. This is achieved by including a pseudo-graph representation in the input embeddings. Performance on downstream tasks by our model underscore promising results of Transformer architectures in addressing the complexities of glycan representation.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Presenter: ~Irene_Rubia-Rodríguez1

Format: Yes, the presenting author will attend in person if this work is accepted to the workshop.

Funding: No, the presenting author of this submission does *not* fall under ICLR’s funding aims, or has sufficient alternate funding.

Submission Number: 53