Symmetries in Weight Space Learning: To Retain or Remove?

Fynn Kiwitt, Behrooz Tahmasebi, Stefanie Jegelka

Keywords: weight space learning, symmetry, invariance, low-rank adaption (LoRA)

Abstract:

Weight-space learning, an emerging paradigm that studies neural networks through their parameter space, has shown promise for tasks ranging from predicting model behavior to addressing privacy risks. An important caveat in weight-space learning is that neural networks admit extensive parameter symmetries: distinct weight configurations can implement the same function. Such symmetries have been studied from multiple angles and play an important role in both theory and practice, including Low-Rank Adaptation (LoRA), a state-of-the-art fine-tuning method for large language models (LLMs) that exhibits scale and rotational invariances.
In this paper, we present a theoretical study of symmetries in weight-space learning and ask: What is the appropriate problem formulation in the presence of symmetries (e.g., those induced by LoRA), and should redundant representations that encode the same end-to-end function be removed? We answer this by showing that whether redundancy matters depends on the target functional of interest. In particular, we prove that end-to-end symmetries (such as those in LoRA) should not always be quotiented out: doing so can compromise universality for classes of weight-space prediction tasks. To our knowledge, this is the first formal identification of this phenomenon, offering principled guidance for the design of weight-space methods across many applications.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 2973