XTransplant: A Probe into the Upper Bound Performance of Multilingual Capability in LLMs via Cross-lingual Transplantation

Yangfan Ye, Xiaocheng Feng, Xiachong Feng, Libo Qin, Yichong Huang, Lei Huang, Weitao Ma, Zhirui Zhang, Yunfei Lu, Xiaohui Yan, Duyu Tang, Dandan Tu, Bing Qin

Keywords: Large language models, Multilingual capability, Feed forward activations, Upper Bound Performance

Abstract:

Current large language models (LLMs) often display significant imbalances in their multilingual capabilities and cultural adaptability, primarily due to their unbalanced and English-centric pretraining data.
For these English-centric LLMs, the disparities between English and non-English languages hinder their ability to utilize their robust English-based capabilities within non-English contexts, while also limiting access to valuable multilingual knowledge derived from non-English "language-specific neurons" within English contexts.
Motivated by this, our work explores the possibility for LLMs to leverage the strengths of both English and non-English languages, aiming to further unlock their multilingual potential.
To this end, we propose a probing method named $\mathcal{X}$Transplant, which directly transplants feed-forward activations from English input to non-English (or from non-English to English) during inference stage, allowing the model to benefit from both English and additional multilingual knowledge.
Through extensive experiments on our pilotsets and representative LLMs across different tasks and languages, we empirically prove that both the multilingual capabilities and cultural adaptability of LLMs hold the potential to be significantly improved by the cross-lingual feed forward transplantation, respectively from $\texttt{En} \rightarrow \texttt{non-En}$ and $\texttt{non-En} \rightarrow \texttt{En}$.
Additionally, we also establish the upper bound performance of LLMs obtained through $\mathcal{X}$Transplant (relative growth of +80\% in multilingual capabilities, +39\% in cultural adaptability), highlighting the underutilization of current LLMs' multilingual potential.
We do hope our further analysis and discussion could suggest promising directions for deeply unlocking the multilingual potential of current English-centric LLMs.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9030