LEARN TO LEARN CONSISTENTLY

Runxi Cheng, Yongxian Wei, Xianglong He, Wanyun Zhu, Songsong Huang, Feng Xiong, Fei Richard Yu, Fei Ma, Zhanqiu Zhang, Chun Yuan

Keywords: meta learning, few-shot learning, meta self-distillation, consistency of learned knowledge

TL;DR: learn the same knowledge From different views of identical image

Abstract:

In the few-shot learning problem, a model trained on a disjoint meta-train dataset
is required to address novel tasks with limited novel examples. A key challenge in
few-shot learning is the model’s propensity to learn biased shortcut features(e.g.,
background, noise, shape, color), which are sufficient to distinguish the few ex-
amples during fast adaptation but lead to poor generalization. In our work, we
observed when the model learns with higher consistency, the model tends to be
less influenced by shortcut features, resulting in better generalization. Based on
the observation, we propose a simple yet effective meta-learning method named
Meta Self-Distillation. By maximizing the consistency of the learned knowledge
during the meta-train phase, the model initialized by our method shows better
generalization in the meta-test phase. Extensive experiments demonstrate that our
method improves the model’s generalization across various few-shot classification
scenarios and enhances the model’s ability to learn consistently.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 155