On the Importance of Trivial Baselines: Re-evaluating LoRA Adapter Transfer for Generative Tasks

Keywords: LoRA, Transfer

Abstract: Modern LLM post-training is becoming increasingly compute-intensive, making the re-alignment and fine-tuning for every new base model iteration computationally unsustainable. While training-free techniques for transferring post-training weights such as LoRA adapters across different base model versions exist, their effectiveness remains under-explored. In this work, we systematically evaluate the efficacy of LoRA weight transfer methods, ranging from simple arithmetic weight transfer to more complex frameworks like CrossLoRA and ProLoRA. We assess performance across a diverse suite of benchmarks, including discriminative tasks (ARC-Easy/Challenge) and multi-token generation (machine translation - WMT19). Our results demonstrate that directly copying LoRA adapters between related base models consistently outperforms more elaborate transfer schemes. However, we identify a significant disparity in transfer robustness: while Multiple Choice Question Answering (MCQA) capabilities are preserved with relative ease, generation performance suffers substantially. Furthermore, we establish that transfer success depends on the similarity between the source and the target base model weights.

Submission Number: 57