SkipCLM: Enhancing Crosslingual Alignment of Decoder Transformer Models via Contrastive Learning and Skip Connection

Download PDF
Open Webpage

Nikita Sushko, Alexander Panchenko, Elena Tutubalina

Published: 2025, Last Modified: 23 Jan 2026NAACL (Student Research Workshop) 2025EveryoneRevisionsBibTeXCC BY-SA 4.0
Abstract: This paper proposes SkipCLM, a novel method for improving multilingual machine translation in Decoder Transformers. We augment contrastive learning for cross-lingual alignment with a trainable skip connection to preserve information crucial for accurate target language generation. Experiments with XGLM-564M on the Flores-101 benchmark demonstrate improved performance, particularly for en-de and en-zh direction translations, compared to direct sequence-to-sequence training and existing contrastive learning methods. Code is available at: https://github.com/s-nlp/skipclm.