How Multilingual are Large Language Models Fine-tuned for Translation?

Aquia Richburg, Marine Carpuat

Research Area: Evaluation, LMs for everyone

Keywords: multilingual machine translation, large language models, translation fine-tuning, multilingual evaluation

TL;DR: Translation fine-tuned large language models improve overall translation quality, even for zero-shot languages, but the impact is uneven depending on the language pairs involved.

Abstract: A new paradigm for machine translation has recently emerged: fine-tuning large language models on parallel text has been shown to outperform dedicated translation systems trained in a supervised fashion on much larger amounts of parallel data (Xu et al. 2024, Alves et al. 2024). However, it remains unclear whether this paradigm can enable massively multilingual machine translation or whether it requires fine-tuning dedicated models for a small number of language pairs. How does translation fine-tuning impact the MT capabilities of LLMs for zero-shot languages, zero-shot language pairs, and translation tasks that do not involve English? To address these questions, we conduct an extensive empirical evaluation of the translation quality of the TOWER family of language models (Alves et al. 2024) on 132 translation tasks from the multi-parallel FLORES data. We find that translation fine-tuning improves translation quality even for zero-shot languages on average, but that the impact is uneven depending on the language pairs involved. These results call for further research to effectively enable massively multilingual translation with LLMs.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html

Author Guide: I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html

Submission Number: 1167