Many-Shot Scaling of In-Context Learning with Self-Generated Demonstrations

ACL ARR 2026 January Submission8335 Authors

Keywords: Unlabeled Data, prompting, scaling

Abstract: The high cost of obtaining high-quality annotated data for in-context learning (ICL) has motivated the development of methods that use self-generated annotations in place of ground truth labels. While these approaches have shown promising results in few-shot settings, they generally do not scale to many-shot scenarios. In this work, we study ICL with self-generated examples using a framework analogous to traditional semi-supervised learning, consisting of annotation generation, demonstration selection, and in-context inference. Within this framework, we propose a simple baseline that outperforms ground truth ICL under zero-shot, few-shot, and many-shot settings. Notably, we observe consistent scaling behaviors with respect to the number of self-annotated demonstrations. To further extract performance from this many-shot capability, we introduce IterPSD, an iterative self-annotation approach that integrates iterative refinement and curriculum pseudo-labeling techniques from semi-supervised learning, yielding up to 6.8% additional gains on classification tasks. Motivated by our baseline and IterPSD results, we demonstrate that semi-supervised ICL offers a promising avenue for future ICL research.

Paper Type: Long

Research Area: Language Models

Research Area Keywords: Unlabeled Data, prompting, scaling

Contribution Types: NLP engineering experiment, Approaches to low-resource settings

Languages Studied: English, Bemba, Tuvan, Venetian, Sardinian, Faroese

Submission Number: 8335