BAT: Backbone Augmented Training for Adaptations

Jae Wan Park, Junhyeok Kim, Hyunah Ko, Youngjun Jun, Seong Jae Hwang

Keywords: Adaptation, Data Selection, Parameter Efficient Tuning, Regularization, Optimization, Dreambooth, LoRA

TL;DR: This paper suggests that backbone augmented training in adaptation surpasses regular adaptation training mathematically and empirically.

Abstract:

Adaptations have enabled efficient training for large backbone models such as
diffusion models for image generation and transformer-based language models.
While various adaptation techniques aim to maximize performance with minimal
computational resources, limited data often leads to challenges like overfitting,
mode collapse, or hallucinations. Recently, a promising solution has emerged in
the form of augmenting adapter datasets using data originally employed to train
backbone models. While this approach has shown potential as a breakthrough, it
often lacks a solid theoretical foundation or well-defined standards for control-
lability. To address these limitations, we establish a comprehensive theoretical
framework for Backbone Augmented Training (BAT). Furthermore, we provide
both theoretical and experimental evidence demonstrating that BAT achieves a
faster convergence rate to optimal adaptation parameters compared to conven-
tional adaptation methods. Our results underscore the potential of backbone aug-
mentation to significantly improve performance, especially when coupled with an
effective and well-designed data selection schema.

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6950