MetaFormer with Holistic Attention Modelling Improves Few-Shot Classification
Keywords: Meta-Learning, Vision Transformers
Abstract: Pre-trained vision transformers have revolutionized few-shot image classification, and it has been recently demonstrated that the previous common practice of meta-learning in synergy with these pre-trained transformers still holds significance and contributes to further advancing their performance. Unfortunately, the majority of working insights such as task conditioning are specifically tailored for convolutional neural networks, thus failing to translate effectively to vision transformers. This work sets out to bridge this gap via a coherent and lightweight framework called MetaFormer, which maintains compatibility with off-the-shelf pre-trained vision transformers. The proposed MetaFormer consists of two attention modules, i.e., the Sample-level Attention Module (SAM) and the Task-level Attention Module (TAM). SAM works in conjunction with the patch-level attention in Transformers to enforce consistency in the attended features across samples within a task, while TAM regularizes learning of the current task with an attended task in the pool. Empirical results on four few-shot learning benchmarks, i.e., miniImageNet, tieredImageNet, CIFAR-FS, and FC100, showcase that our approach achieves the new state-of-the-art at a very modest increase in computational overhead. Furthermore, our approach excels in cross-domain task generalization scenarios.
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 9345
Loading