Learning to Learn with Contrastive Meta-Objective

Shiguang Wu, Yaqing Wang, Yatao Bian, Quanming Yao

Keywords: Meta-Learning, Contrastive Learning

Abstract:

Meta-learning enables learning systems to adapt quickly to new tasks, similar to humans.
Different meta-learning approaches all work under/with the mini-batch episodic training framework. Such framework naturally gives the information about task identity, which can serve as additional supervision for meta-training to improve generalizability. We propose to exploit task identity as additional supervision in meta-training, inspired by the alignment and discrimination ability which is is intrinsic in human's fast learning.
This is achieved by contrasting what meta-learners learn, i.e., model representations.
The proposed ConML is evaluating and optimizing the contrastive meta-objective under a problem- and learner-agnostic meta-training framework.
We demonstrate that ConML integrates seamlessly with existing meta-learners, as well as in-context learning models, and brings significant boost in performance with small implementation cost.

Primary Area: General machine learning (supervised, unsupervised, online, active, etc.)

Submission Number: 6278