Unveiling Multilingual Dynamics: Aligning Representations through Internal Layer Probing

ACL ARR 2025 February Submission5923 Authors

Abstract: In this paper, we investigate how large language models (LLMs) process multilingual tokens within their layer representations—an open question despite significant advancements in the field. Using simple probing techniques through representation engineering, we demonstrate that steering a single model layer can notably enhance performance. Our analysis shows that this approach achieves results comparable to translation baselines and surpasses state-of-the-art prompt optimization methods. Additionally, we highlight how advanced techniques like supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) improve multilingual capabilities by altering representation spaces. We further illustrate how these methods align with our approach to reshaping LLM layer representations.

Paper Type: Long

Research Area: Syntax: Tagging, Chunking and Parsing

Research Area Keywords: Multilingual Processing , Representation Engineering , Cross-lingual Understanding , Multilingual Performance Optimization Model Steering Techniques

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Approaches to low-resource settings, Publicly available software and/or pre-trained models, Data analysis

Languages Studied: English ,Spanish, French, Russian, German, Japanese, Chinese, Turkish, Arabic, Vietnamese, Hindi, Greek, Indonesian, Italian, Portuguese.

Submission Number: 5923