Latent Merging: Dynamic and Reversible Composition of Large Language Models

Jea Sung Kim, Suan Lee

Keywords: Latent merging, Model merging, Hidden representations, Reversible composition, Layer-wise control, Large language models

TL;DR: A framework for merging large language models at inference time by composing latent representations instead of weights, enabling reversible, conditional, and layer-wise control without modifying parameters

Abstract: Weight merging is a common way to combine large language models, but its static and irreversible nature limits controllability and can destabilize behavior. We propose Latent Merging, which composes models in the hidden-representation space to enable dynamic, reversible, and layer-wise control without modifying weights. We unify classic operators—linear/spherical interpolation, and regularized means—under a single operator view and extend them from parameters to latents. We derive local second-order bounds on loss change that account for RMSNorm nonlinearity and head mismatch, yielding practical guidance (merge later; align heads) and stability guarantees. In data-free evaluation on Qwen2.5-7B-Instruct and its fine-tuned derivative, Latent Merging consistently surpasses weight merging on JudgeBench across reasoning, knowledge, mathematics, and coding; for example, SLERP attains 74.8 overall vs. 25.3 for weight merging. Representation analyses show stronger semantic preservation (cosine/CKA > 0.8), and layer-wise studies indicate that higher mixing ratios ($\alpha \approx 0.75$) in deeper layers work best while remaining bounded by source-model capacity. Latent Merging reframes model composition as controlling states rather than rewriting weights, offering a practical, theory-grounded path to controllable and interpretable LLM integration.

Primary Area: generative models

Submission Number: 9102