Advantages, Risks and Insights from Comparing In-Context Learning Models with Typical Meta-Learners

ICLR 2025 Conference Submission13664 Authors

Keywords: In-Context Learning, Meta-Learning

Abstract:

We investigate in-context learning (ICL) models from the perspective of learning to learn.
Unlike existing studies that focus on identifying the specific learning algorithms that ICL models learn, we compare ICL models with typical meta-learners to understand their superior performance.
We theoretically prove the expressiveness of ICL models as learning algorithms and examine their learnability and generalizability across extensive settings. Our findings demonstrate that ICL with transformers can effectively learn data-dependent optimal learning algorithms within an inclusive space that encompasses gradient-based, metric-based, and amortization-based meta-learners.
However, we identify generalizability as a critical issue, as the learned algorithms may implicitly fit the training distribution rather than embodying explicit learning processes. Based on this understanding, we propose transferring deep learning techniques, widely studied in supervised learning, to meta-learning to address these common challenges. As examples, we implement meta-level meta-learning for domain adaptability with limited data and meta-level curriculum learning for accelerated convergence during pre-training, demonstrating their empirical effectiveness.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 13664