Mixup Model Merge: A Geometric Exploration of Task-Vector Space for Decoupled Model Merging

ACL ARR 2026 January Submission2098 Authors

Keywords: Model Merging, Large Language Models, Task Vectors, Geometric Exploration, Parameter Interpolation

Abstract: Model merging integrates distinct task-specific capabilities into a unified model without the cost of retraining. We observe that prevailing methods largely focus on resolving parameter-level conflicts while overlooking a macroscopic geometric constraint: they typically assign uniform weights that lock the merging direction, allowing variation only in magnitude. We posit that this rigid constraint prevents the discovery of optimal solutions in the parameter space. In this work, we introduce Mixup Model Merge (M$^3$), a framework designed to navigate this unexplored geometry by decoupling the exploration of merging direction (via interpolation coefficients) and magnitude (via a scaling factor). By utilizing randomized linear interpolation, M$^3$ systematically probes the continuous task-vector space to identify low-loss merging configurations. Empirical results on three task-specific LLMs reveal that this simple geometric exploration significantly outperforms complex conflict-resolution baselines. Moreover, M$^3$ demonstrates superior generalization, substantially enhancing out-of-distribution and adversarial robustness on benchmarks such as LiveBench and PromptBench. Notably, M$^3$ is orthogonal to sparsification techniques like DARE, unlocking further performance improvements when combined.

Paper Type: Long

Research Area: Machine Learning for NLP

Research Area Keywords: optimization methods, representation learning, generalization, multi-task learning, model merging, task vectors

Contribution Types: NLP engineering experiment, Approaches low compute settings-efficiency, Publicly available software and/or pre-trained models

Languages Studied: English

Submission Number: 2098