Towards Modular Fine-tuning of LLM-based Multilingual Neural Machine Translation

ACL ARR 2025 July Submission1011 Authors

Abstract: Multilingual Fine-tuning of Large Language Models (LLMs) has achieved great advancements in machine translation. However, existing research only focuses on a single-shot fine-tuning process with fixed training data, often lacking adaptability, where introducing new languages will influence the performance of existing ones. In this study, we propose a modular fine-tuning pipeline that enables dynamic language support for LLMs. Instead of directly fine-tuning on all languages, our approach first trains English-centric LoRA adapters for each input and output language separately, and then merges the corresponding adapters' parameters without any training during translation. Experiments on 12 translation directions involving four low-resource and less-supported languages show that modular fine-tuning achieves up to $86\%$ performance of traditional multi-parallel full-parameter fine-tuning, while using only $0.1\%$ trainable parameters and relying solely on English-centric data. Furthermore, we perform a comprehensive analysis about the merging ratio, when to merge, and the rationale for using English as a bridge language via Bayesian Optimization and logit lens.

Paper Type: Long

Research Area: Machine Translation

Research Area Keywords: Machine Translation,Multilingualism and Cross-Lingual NLP,Efficient/Low-Resource Methods for NLP

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency

Languages Studied: English

Submission Number: 1011