Representative Demonstration Selection for In-Context Learning with Two-Stage Determinantal Point Process

Zhao Yang, Yuanzhe Zhang, Dianbo Sui, Cao Liu, Jun Zhao, Kang Liu

Submission Type: Regular Long Paper

Submission Track: Language Modeling and Analysis of Language Models

Keywords: In-Context Learning, Representative Subset Selection, Determinantal Point Process

TL;DR: This paper invertigates how to select representative demonstrations for in-context learning and propose a two-stage DPP selection frame to solve this problem.

Abstract: Although In-Context Learning has proven effective across a broad array of tasks, its efficiency is noticeably influenced by the selection of demonstrations. Existing methods tend to select different demonstrations for each test instance, which is time-consuming and poses limitations in practical scenarios. Therefore, this study aims to address the challenge of selecting a representative subset of in-context demonstrations that can effectively prompt different test instances in a specific task. We propose that this representative subset should be of high quality and diversity. Our empirical analyses confirm that demonstrations that meet these criteria can indeed bolster model performance. To satisfy these criteria, this paper further introduces a two-stage Determinantal Point Process (DPP) method designed to incorporate both quality and diversity in the process of demonstration selection, thereby obtaining representative in-context demonstrations. Through comprehensive experimentation, we have confirmed the efficacy of our proposed method, paving the way for more practical and effective In-Context Learning.

Submission Number: 2871