Exploring Translation Mechanism of Large Language Models

ACL ARR 2025 February Submission4827 Authors

Abstract:

Large language models (LLMs) have succeeded remarkably in multilingual translation tasks.
However, the inherent translation mechanisms of LLMs remain poorly understood, largely due to sophisticated architectures and vast parameter scales.
In response to this issue, this study explores the translation mechanism of LLM from the perspective of computational components (e.g., attention heads and MLPs).
Path patching is utilized to explore causal relationships between components, detecting those crucial for translation tasks and subsequently analyzing their behavioral patterns in human-interpretable terms.
Comprehensive analysis reveals that translation is predominantly facilitated by a sparse subset of specialized attention heads (less than 5\%), which extract source language, indicator, and positional features. MLPs subsequently integrate and process these features by transiting towards English-centric latent representations.
Notably, building on the above findings, targeted fine-tuning of only 64 heads achieves translation improvement comparable to full-parameter tuning while preserving general capabilities.

Paper Type: Long

Research Area: Interpretability and Analysis of Models for NLP

Research Area Keywords: Interpretability and Analysis of Models for NLP, Machine Translation, Multilingualism and Cross-Lingual NLP,

Contribution Types: Model analysis & interpretability

Languages Studied: Chinese, English, Russian, German, French, Czech, Japanese

Submission Number: 4827