Efficient Ensembling Improves Training Data Attribution

TMLR Paper5806 Authors

Abstract:

Training data attribution (TDA) methods aim to quantify the influence of individual training data points on the model predictions, with broad applications in data-centric AI, such as mislabel detection, data selection, and copyright compensation. However, existing methods in this field, which can be categorized as retraining-based and gradient-based, have struggled with the trade-off between computational efficiency and attribution efficacy. Retraining-based methods can accurately attribute complex non-convex models but are computationally prohibitive, while gradient-based methods are efficient but often fail for non-convex models. Recent research has shown that augmenting gradient-based methods with ensembles of multiple independently trained models can achieve significantly better attribution efficacy. However, this approach remains impractical for very large-scale applications.

In this work, we discover that expensive, fully independent training is unnecessary for ensembling the gradient-based methods, and we propose two efficient ensemble strategies, DROPOUT ENSEMBLE and LORA ENSEMBLE, alternative to naive independent ensemble. These strategies significantly reduce training time (up to 80%), serving time (up to 60%), and space cost (up to 80%) while maintaining similar attribution efficacy to the naive independent ensemble. Our extensive experimental results demonstrate that the proposed strategies are effective across multiple TDA methods on diverse datasets and models, including various generative settings, significantly advancing the Pareto frontier of TDA methods with better computational efficiency and attribution efficacy. We conduct a theoretical analysis that provides insights into the success of our empirical findings.

Submission Length: Regular submission (no more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=PjldrbYqeu

Changes Since Last Submission:

- We add clear descriptions in text in Sections 4.2 & 4.3 as well as annotations in Figures 3 & 6 to clearly present how the efficiency gain claimed is achieved. All of the cost metrics (training, serving, and storage) are analyzed in Sections 4.2 & 4.3. The figures (e.g., Figure 5) help the understanding of these cost metrics are also upated.

- We add empirical studies (including the empirical computation of covariance) to connect the theoretical analysis with Dropout Ensemble and LoRA Ensemble performance compared with Naive Ensemble. This also shows that the theoretical analysis, empirical computation of covariance, and efficacy performance are aligned.

- A reorganization of appendix sections with a content list added in front of them for better clarity. Some non-referenced tables and figures are also fixed.

- Include large experiment settings' results in Section 4.4 as well as their detailed experiment settings.

- Include a discussion of the comparison between the two efficient ensemble methods we proposed in Section 4.5.

Assigned Action Editor: ~Jasper_Snoek1

Submission Number: 5806