Learning to Learn with Contrastive Meta-Objective

Shiguang Wu, Yatao Bian, Quanming Yao

Keywords: meta learning, contrastive learning, few-shot learning, i n-context learning

TL;DR: We propose contrastive meta-objective as additional supervision for meta-training, that can significantly and universally improve meta-learning and in-context learning performance with cheap implementation.

Abstract:

We propose a contrastive meta-objective to enable meta-learners to emulate human-like rapid learning capability through enhanced alignment and discrimination. Our proposed approach, dubbed ConML, exploits task identity as additional supervision signal for meta-training, benefiting meta-learner's fast-adaptation and task-level generalization abilities. This is achieved by contrasting the outputs of meta-learner, i.e, performing contrastive learning in the model space.
Specifically, we introduce metrics to minimize the inner-task distance, i.e., the distance among models learned on varying data subsets of the same task, while maximizing the inter-task distance among models derived from distinct tasks.
ConML distinguishes itself through versatility and efficiency, seamlessly integrating
with episodic meta-training methods and the in-context learning of large language models (LLMs).
We apply ConML to representative meta-learning algorithms spanning optimization-, metric-, and amortization-based approaches, and show that ConML can universally and significantly improve conventional meta-learning and in-context learning.

Primary Area: Optimization for deep networks

Submission Number: 7145