ImageNet-RIB Benchmark: Large Pre-Training Datasets Don't Guarantee Robustness after Fine-Tuning
Keywords: robust fine-tuning, robustness, transfer learning, representation learning, continual learning
TL;DR: We introduce ImageNet-RIB, a robust fine-tuning benchmark showing that pre-training on larger datasets does not guarantee better robustness to OOD data after fine-tuning than smaller datasets, while continual learning helps robustness.
Abstract: Highly performant large-scale pre-trained models promise to also provide a valuable foundation for learning specialized tasks, by fine-tuning the model to the desired task.
By starting from a good general-purpose model, the goal is to achieve both specialization in the target task and maintain robustness.
To assess the robustness of models to out-of-distribution samples after fine-tuning on downstream datasets,
we introduce a new robust fine-tuning benchmark, ImageNet-RIB (Robustness Inheritance Benchmark).
The benchmark consists of a set of related but distinct specialized (downstream) tasks; pre-trained models are fine-tuned on one task in the set and their robustness is assessed on the rest, iterating across all tasks for fine-tuning and assessment.
We find that the continual learning methods, EWC and LwF maintain robustness after fine-tuning though fine-tuning generally does reduce performance on generalization to related downstream tasks across models.
Not surprisingly, models pre-trained on large and rich datasets exhibit higher initial robustness across datasets and suffer more pronounced degradation during fine-tuning.
The distance between the pre-training and downstream datasets, measured by optimal transport, predicts this performance degradation on the pre-training dataset.
However, counterintuitively, model robustness after fine-tuning on related downstream tasks is the worst when the pre-training dataset is the richest and the most diverse.
This suggests that starting with the strongest foundation model is not necessarily the best approach for performance on specialist tasks.
The benchmark thus offers key insights for developing more resilient fine-tuning strategies and building robust machine learning models.
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 566
Loading