Meta-Learning with Personalized Learning Rates for Rapid Task Mastery

Maofa Wang, Zhixiong Leng, Quan Wan, Yanlin Xu, Bingchen Yan, Zhenbing Liu

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Few-shot Learning, Meta-learning, Adaptive Learning Rate, Rapid Adaptation, Information Loss

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Traditional meta-learning approaches primarily focus on the generalization ability of models across unfamiliar tasks. These methods typically involve fine-tuning the model in the outer loop to perform well on new tasks. While this is valuable for enabling models to adapt to various tasks, it may overlook the details of rapid adaptation within tasks. During rapid adaptation, the same task may exhibit entirely different data distributions, features, and patterns in different training phases, making it exceptionally challenging to determine an appropriate learning rate. Consequently, conventional meta-learning methods often employ fixed learning rates or simple learning rate strategies, overlooking the dynamic nature within tasks. In this paper, we propose an Meta-Learning with Personalized Learning Rates (MLPLR) approach. Specifically, we adaptively generate negatively correlated learning rates by evaluating the information loss between predicted values and ground truth. When the information loss is low, indicating the model's strong performance on the current task, we can increase the learning rate to expedite the learning process. This aids in faster convergence and adapting to specific patterns and features within tasks. Conversely, when the information loss is high, indicating poor model performance on the current task, we reduce the learning rate to ensure more stable and gradual parameter updates, thereby mitigating overfitting. Extensive experiments and analyses demonstrate that our approach enhances the performance of various meta-learning models in the contexts of few-shot classification, few-shot fine-grained classification, and cross-domain few-shot classification.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3274