Inspiring Out-of-distribution Generalization of In-Context Learning via Contrastive Demonstrations

Anonymous

Abstract: Although existing demonstration construction methods have significantly improved the performance of In-Context Learning (ICL), these unfortunately only focused on the in-distribution settings that the selected demonstrations should have the same distribution with testing data. However, the out-of-distribution (OOD) settings are more commonly encountered in real scenarios, but ignored in the age of large language models. This paper first investigates the performance of existing ICL demonstration construction methods in OOD settings and verifies their failures. Moreover, this paper proposes contrastive demonstrations that combine a demonstration with its counterfactual, where a rationale-guided counterfactual generation method is proposed to generate higher-quality counterfactual data. Extensive experiments validate the effectiveness of our proposed method and the contrastive demonstrations can help the model better identify the essence of the task, thus achieving OOD generalization.

Paper Type: short

Research Area: NLP Applications

Contribution Types: NLP engineering experiment

Languages Studied: English