MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning

Zihan Chen, Song Wang, Zhen Tan, Jundong Li, Cong Shen

Abstract:

In-Context Learning (ICL) empowers Large Language Models (LLMs) to tackle diverse tasks by incorporating multiple input-output examples, known as demonstrations, into the input of LLMs. More recently, advancements in the expanded context windows of LLMs have led to many-shot ICL, which uses hundreds of demonstrations and outperforms few-shot ICL, which relies on fewer examples. However, this approach is often hindered by the high cost of obtaining large amounts of labeled data. To address this challenge, we propose **M**any-Shot **A**daptive **P**seudo-**L**ab**E**ling, namely **MAPLE**, a novel influence-based many-shot ICL framework that utilizes pseudo-labeled samples to compensate for the lack of label information.
We first identify a subset of impactful unlabeled samples and perform pseudo-labeling on them by querying LLMs. These pseudo-labeled samples are then adaptively selected and tailored to each test query as input to improve the performance of many-shot ICL, without significant labeling costs.
Extensive experiments on real-world datasets demonstrate the effectiveness of our framework, showcasing its ability to enhance LLM adaptability and performance with limited labeled data. Our code is provided at https://github.com/Chen-1031/MAPLE_ICL.

Lay Summary:

Large language models like ChatGPT can solve tasks such as summarizing text or answering questions by looking at a few examples—this is called in-context learning (ICL). When given more examples (many-shot ICL), models usually perform better, but collecting many labeled examples can be expensive and time-consuming.

Our work proposes a way to improve many-shot ICL without needing so many human-labeled examples. Instead, we identify the most useful unlabeled examples, ask the model itself to infer their labels (a technique called pseudo-labeling), and then carefully choose which of these examples to show the model when it's solving new tasks.
We call this method MAPLE. It builds a network connecting labeled and unlabeled data to determine which samples are most helpful. MAPLE then adaptively selects the most relevant examples for each question.
This approach significantly boosts model performance across a wide range of tasks while keeping labeling costs low, making powerful AI more accessible and practical for real-world use.

Primary Area: Deep Learning->Large Language Models

Keywords: In-context learning, Many-shot In-context learning, Large language models

Submission Number: 14586