Model Merging with Functional Dual Anchors

ICLR 2026 Conference Submission884 Authors

Keywords: Model Merging, Knowledge Transfer, Multi-task Learning

Abstract: Model merging is an efficient post-training strategy for integrating knowledge from multiple finetuned checkpoints of a shared foundation model. Existing methods operate in the parameter space, combining task vectors to mitigate conflicts, but remain constrained by parameter inconsistencies. We propose Functional Dual Anchors (FDAs), a framework that instead models the input–representation space. FDAs are synthetic inputs whose induced gradients align with task vectors, capturing task-specific functional shifts relative to the pretrained model. This perspective bridges joint multi-task training and post-hoc merging, offering both robustness and flexibility. We further introduce a principled initialization scheme and show that FDAs are complementary to parameter-centric model merging. Comprehensive experiments demonstrate the effectiveness of FDAs in model merging.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 884