Automatic Combination of Sample Selection Strategies for Few-Shot Learning

Branislav Pecher, Ivan Srba, Maria Bielikova, Joaquin Vanschoren

Keywords: sample selection, few-shot learning, in-context learning, large language models, meta-learning, few-shot fine-tuning, data-centric

TL;DR: We investigate the impact of curating a small set of informative and high-quality samples on success of different few-shot learning approaches and propose a novel method to select such samples based on their complementary properties

Abstract: In few-shot learning, such as meta-learning, few-shot fine-tuning or in-context learning, the selection of samples has a significant impact on the performance of the trained model. Although many sample selection strategies are employed and evaluated in typical supervised settings, their impact on the performance of few-shot learning is largely unknown. In this paper, we investigate the impact of 20 sample selection strategies on the performance of 5 representative few-shot learning approaches over 8 image and 6 text datasets. We propose a new method for Automatic Combination of SamplE Selection Strategies (ACSESS), to leverage the strengths and complementarity of the individual strategies in order to select more impactful samples. The experimental results show that our method consistently outperforms all individual selection strategies. We also show that the majority of existing strategies strongly depend on modality, dataset characteristics and few-shot learning approach, while improving performance especially on imbalanced and noisy datasets. Lastly, we show that sample selection strategies work well even on smaller datasets and provide larger benefit when selecting a lower number of shots, while frequently regressing to random selection with higher numbers of shots.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9388