ProtoReg: Prioritizing Discriminative Information for Fine-grained Transfer Learning
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Transfer learning, fine-tuning, regularization
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
TL;DR: We propose a simple yet effective method that utilizes adaptively evolving class prototypes to capture fine-grained information.
Abstract: Transfer learning leverages a pre-trained model with rich features to fine-tune it for downstream tasks, thereby improving generalization performance. However, we point out the "granularity gap" in fine-grained transfer learning, a mismatch between the level of information learned by a pre-trained model and the semantic details required for a fine-grained downstream task. Under these circumstances, excessive non-discriminative information can hinder the sufficient learning of discriminative semantic details. In this study, we address this issue by establishing class-discriminative prototypes and refining the prototypes to gradually encapsulate more fine-grained semantic details, while explicitly aggregating each feature with the corresponding prototype. This approach allows the model to prioritize fine-grained discriminative information, even when the pre-trained model contains excessive non-discriminative information due to the granularity gap. Our proposed simple yet effective method, ProtoReg, significantly outperforms other transfer learning methods in fine-grained classification benchmarks with an average performance improvement of 6.4\% compared to standard fine-tuning. Particularly in limited data scenarios using only 15\% of the training data, ProtoReg achieves an even more substantial average improvement of 13.4\%. Furthermore, ProtoReg demonstrates robustness to shortcut learning when evaluated on out-of-distribution data.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 7226
Loading