Beyond Conflict: Subspace-Alignment as the Missing Piece of Model Merging

Yayuan Li, Ze Peng, Jian Zhang, Jintao Guo, Yue Duan, Yinghuan Shi

Keywords: Model Merging, Transfer Learning, Multi-task Learning

Abstract:

Model merging integrates task knowledge by combining task vectors (differences between pre-trained and fine-tuned weights) across models for both vision and large language models (LLMs).
Existing methods improve merging performance solely by mitigating conflicts between task vectors. However, we find that conflict alone is not the full story:
when conflict-oriented remedies are applied, severe interference still persists. This observation motivates a novel perspective: analysing the interference from the standpoint of alignment.
Experiments reveal that task vectors exhibit high alignment in subspaces with large singular values. After merging, these aligned subspaces gather more singular values and produce activations with higher magnitude compared to others. The resulting spectral imbalance substantially degrades model performance.
Inspired by this, in contrast to previous methods that focus solely on conflicts, we propose the Subspace-Alignment Aware Merging (AlignMerge), which quantifies alignment by projecting task vectors onto the shared singular subspaces of the merged task vector and attenuates overly aligned components.
AlignMerge is training-free and requires no auxiliary data. Across vision and language benchmarks, it achieves state-of-the-art performance and narrows the
gap to traditional multi-task learning to 3.6\%.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 12914