Mechanisms are Transferable: Data-Efficient Low-Resource Adaptation via Circuit-Targeted Supervised Fine-Tuning

ACL ARR 2026 January Submission8755 Authors

Keywords: Mechanistic Interpretability, Circuit Discovery, Circuit-Targeted Fine-Tuning, Low-Resource languages, Cross-Lingual Transfer, Data-Efficient Adaptation

Abstract: Adapting LLMs to low-resource languages is difficult: labeled data is scarce, full-model fine-tuning is unstable, and continued cross-lingual tuning can cause catastrophic forgetting. We propose Circuit-Targeted Supervised Fine-Tuning (CT-SFT): a counterfactual-free adaptation of CD-T (Contextual Decomposition Transformer) that uses a label-balanced mean baseline and task-directional relevance scoring to identify a sparse set of task-relevant attention heads in a proxy-language checkpoint, then transfer learns to a target language by updating only those heads (plus LayerNorm) via head-level gradient masking. Across NusaX-Senti and XNLI, CT-SFT improves cross-lingual accuracy over continued full fine-tuning while updating only a small subset of model parameters. We find an editing-preserving trade-off: harder transfers favor editing circuit heads, while easier transfers often favor near-zero (i.e., low-relevance heads) updates, preserving the source mechanism. CT-SFT also substantially reduces catastrophic forgetting, preserving proxy/source-language competence during transfer.

Paper Type: Long

Research Area: Low-resource Methods for NLP

Research Area Keywords: parameter-efficient-training, data-efficient training, NLP in resource-constrained settings

Contribution Types: Model analysis & interpretability, Approaches to low-resource settings, Approaches low compute settings-efficiency

Languages Studied: Indonesian, Buginese, Minangkabau, Javanese, Acehnese, English, Spanish, Thai, Vietnamese, Chinese

Submission Number: 8755