Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature

ICLR 2026 Conference Submission21000 Authors

Keywords: task arithmetic, model editing, task vector

TL;DR: We tackle cross-task interference in Task Arithmetic with a data-free approach. By viewing drift as curvature approximation and applying K-FAC, we achieve scalable task disentanglement and improved benchmark performance.

Abstract: Task Arithmetic (TA) provides a modular and scalable way to adapt foundation models. Combining multiple task vectors, however, can lead to cross-task interference, causing representation drift and degraded performance. Representation drift regularization provides a natural remedy to disentangle task vectors, but existing approaches typically require external task data, which conflicts with TA’s modularity and availability constraints like privacy concerns. We propose a data-free approach by framing representation drift regularization as a curvature matrix approximation problem. This allows us leverage well-established techniques; in particular, we adopt Kronecker-Factored Approximate Curvature (KFAC) to obtain practical regularizers. Our method is data-free, has constant complexity with respect to the number of tasks, and improves performance on TA benchmarks.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 21000