Audit Before You Merge: Provenance, Probing, and Continual LoRA Composition

Lenny Aharon, Neta Glazer, Lior Aharon

Keywords: LoRA merging; catastrophic forgetting; continual learning; evaluation methodology

TL;DR: Public LoRA adapters often fail basic provenance checks, inflating measured forgetting by ~10pp; on audited adapters, Task Arithmetic already matches the unmerged baseline.

Abstract:

Post-hoc merging of public LoRA adapters is an attractive
gradient-free path to continual adaptation, but reliable measurement
of catastrophic forgetting in this setting depends on infrastructure
that prior work has not addressed. We treat each adapter merge as a
sequential task in a continual-learning setting, with the
instruction-tuned base model's capabilities as Task~0, and study
forgetting on Llama-3.1-8B-Instruct. Public adapter hubs contain
modules that fail basic provenance checks: silent base-version
mismatches and near-zero delta norms whose inclusion inflates
measured interference by ${\sim}10$pp. We propose a two-check audit
(base-model match, non-trivial delta norm) and a leakage-free
probing protocol as preconditions for any forgetting study in this
setting. On an audited 4-adapter set, naive merging causes $-10.5$pp
on GSM8K and $-18.3$pp on HumanEval, while Task Arithmetic at
$\lambda{=}0.5$ sits within a standard error of the unmerged
baseline. A coordinate search including \emph{negative} coefficients
identifies the math adapter as probe-optimal at $\lambda{=}-0.5$,
outside the reach of positive-only mergers, suggesting forgetting
has structure that negation-aware merging may exploit. The same
protocol replicates on Mistral-7B-Instruct-v0.3, where 2 of 4
candidate adapters fail the audit and the audited merge reproduces
the same merge-method ranking on GSM8K, supporting that our findings
are not Llama-specific.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 23