Efficient Alignment: Improving Multilingual Models Through Lightweight Steering

ACL ARR 2025 July Submission1132 Authors

Abstract: This paper investigates how Large Language Models (LLMs) represent non-English tokens—a question that remains underexplored despite recent progress. We propose a lightweight intervention method using \textit{representation steering}, where a learned vector is added to the residual stream at a single model layer to enhance multilingual performance. Through extensive experiments across seven competitive baselines—including prompt optimization, supervised fine-tuning SFT, in-context learning, cross-lingual transfer, and translation-based methods—we show that our approach consistently outperforms most alternativ es. In particular, it achieves performance on par with production-grade translation systems while requiring far fewer resources. We further explore the complementarity between our method and \SFT, demonstrating that steering offers a direct, efficient way to realign internal representations. These findings underscore the potential of activation-level interventions as a powerful tool for improving the multilingual capabilities of LLMs.

Paper Type: Long

Research Area: Multilingualism and Cross-Lingual NLP

Research Area Keywords: code-switching, multilingualism, language contact, language change, linguistic variation, cross-lingual transfer, multilingual pre-training, less-resourced languages, endangered languages, indigenous languages, multilingual benchmarks.

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency, Publicly available software and/or pre-trained models

Languages Studied: English ,Spanish, French, Russian, German, Japanese, Chinese, Turkish, Arabic, Vietnamese, Hindi, Greek, Indonesian, Italian, Portuguese.

Previous URL: https://openreview.net/forum?id=SiL1ntfqu9

Reassignment Request Area Chair: Yes, I want a different area chair for our submission

Reassignment Request Reviewers: Yes, I want a different set of reviewers

Justification For Not Keeping Action Editor Or Reviewers: While we appreciate the feedback from the previous reviewers, their comments largely focused on minor issues such as typos and language clarity, all of which have been addressed in the revised manuscript. We also proactively addressed the reviewers' concerns, despite not receiving any substantive feedback following our rebuttal. Unfortunately, the absence of further reviewer input limited our ability to incorporate more targeted or insightful improvements

A1 Limitations Section: This paper has a limitations section.

A2 Potential Risks: Yes

A2 Elaboration: Ethics Statement

B Use Or Create Scientific Artifacts: Yes

B1 Cite Creators Of Artifacts: Yes

B1 Elaboration: 4,5,6

B2 Discuss The License For Artifacts: N/A

B2 Elaboration: N/A

B3 Artifact Use Consistent With Intended Use: No

B3 Elaboration: N/A

B4 Data Contains Personally Identifying Info Or Offensive Content: No

B5 Documentation Of Artifacts: No

B6 Statistics For Data: Yes

B6 Elaboration: 4

C Computational Experiments: Yes

C1 Model Size And Budget: Yes

C1 Elaboration: 4

C2 Experimental Setup And Hyperparameters: Yes

C2 Elaboration: 4

C3 Descriptive Statistics: Yes

C3 Elaboration: 5,6,7

C4 Parameters For Packages: N/A

C4 Elaboration: 4

D Human Subjects Including Annotators: No

D1 Instructions Given To Participants: No

D2 Recruitment And Payment: No

D3 Data Consent: N/A

D4 Ethics Review Board Approval: N/A

D5 Characteristics Of Annotators: N/A

E Ai Assistants In Research Or Writing: Yes

E1 Information About Use Of Ai Assistants: Yes

E1 Elaboration: We used Ai assistants for writing

Author Submission Checklist: yes

Submission Number: 1132