Interfering with Interference: Blind Shuffling and Superposition for Better Multi-Model Compression

Hangyu Zhou, Aaron Gokaslan, Volodymyr Kuleshov, Bharath Hariharan

Keywords: Task Arithmetic, Superposition, Model Merging, Multi-model Compression, Model Serving

TL;DR: We present two complementary random mechanisms to significantly reduce interference when eliminating cross-model redundancy for efficient multi-model serving: Layer Shuffling and Task Vector Superposition.

Abstract: We present two complementary random mechanisms to significantly reduce interference when eliminating cross-model redundancy for efficient multi-model serving: _Layer Shuffling_ and _Task Vector Superposition_. They work together to increase the orthogonality among interfering task vectors, forcing them into self-destruction without requiring any post-training learning or optimization. _Layer Shuffling_ randomly reorders layers of each individual models to reduce the alignment between interfering task vectors. While _Task Vector Superposition_ leverages random orthogonal transformations to decorrelate task vectors further. Together, these techniques drastically minimize interference, yielding improved performance across multiple tasks with effectively zero incremental memory cost when incorporating new models. Their data and model-independent nature also allows for seamless on-the-fly addition or removal of models, without requiring any re-computation, making them highly practical for real-world deployment scenarios.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8535