A Unified SVD Perspective: Deconstructing, Evaluating, and Improving Model Merging with Ortho-Merge

Wei Ruan, Tianming Liu, Jin Lu

Keywords: Model Merging, Singular Value Decomposition (SVD), Orthogonalization, Cross-task Interference, Training-free

Abstract: Model merging is a powerful training-free technique for integrating the capabilities of multiple fine-tuned models, yet prevailing approaches—parameter-statistical (e.g., Average, TIES, DARE) and spectral/SVD-based (e.g., iso_c, KnOTS)—arise from disparate philosophies without a unifying account. We present a unified SVD-centric framework grounded in four principles—energy preservation, cross-task interference, spectral entropy, and information loss—that provides a consistent lens for analyzing merging algorithms. Guided by this framework, we introduce ORTHO-MERGE, a sign-aware deconflict-then-harmonize method. For each layer and task vector, we perform SVD and use signed similarities between leading singular directions to detect both redundant (> τ) and oppositional (< −τ) interference across tasks. The weaker singular component in each interfering pair is removed from its source task vector; the deconflicted vectors are then aggregated and harmonized via iso_c-style spectral averaging (SVD with mean-singular-value equalization). This training- and data-free pipeline resolves geometric conflicts before aggregation and controls the merged spectrum, preserving informative mid-rank structure while avoiding over-flattening. Across three CLIP backbones (ViT-B/32, ViT-B/16, ViT-L/14) and task suites of size 8/14/20, ORTHO-MERGE achieves state-of-the-art or competitive results on both average absolute and normalized accuracy. Spectrum diagnostics further show reduced spectral entropy and lower information loss, aligning the observed gains with our framework.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2026/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 23155