Weak-to-Strong Enhanced Vision Model
Keywords: weak-to-strong enhancement, knowledge distillation
Abstract: Recent advancements in large language and vision models have demonstrated extraordinary capabilities, driving researchers to train increasingly larger models in pursuit of even greater performance. However, smaller, easier-to-train models often exist prior to these larger models. In this paper, we explore how to effectively leverage these smaller, weaker models to assist in training larger, stronger models. Specifically, we investigate the concept of weak-to-strong knowledge distillation within vision models, where a weaker model supervises a stronger one, aiming to enhance the latter’s performance beyond the limitations of the former.
To this end, we introduce a novel, adaptively adjustable loss function that dynamically calibrates the weaker model’s supervision based on the discrepancy between soft labels and hard labels. This dynamic adjustment allows the weaker model to provide more effective guidance during training.
Our comprehensive experiments span various scenarios, including few-shot learning, transfer learning, noisy label learning, and common knowledge distillation settings. The results are compelling: our approach not only surpasses benchmarks set by strong-to-strong distillation but also exceeds the performance of fine-tuning strong models on full datasets. These findings highlight the significant potential of weak-to-strong distillation, demonstrating its ability to substantially enhance vision model performance. Code will be released.
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 775
Loading