Distilling Examples into Task Instructions: Enhanced In-Context Learning for Long B2B Conversations

ACL ARR 2025 July Submission91 Authors

Abstract:

In-context learning has emerged as the standard method for text classification with large language models (LLMs), particularly excelling in scenarios with limited annotated training data. However, their performance on long-context inputs such as business-to-business (B2B) conversations remains underexplored.

We introduce the Call Playbook dataset: five novel classification tasks derived from real-world B2B conversations targeting core sales concepts. Analysis reveals that traditional few-shot learning suffers from performance degradation and prohibitive computational costs in these long-context settings.

To address these challenges, we propose knowledge extraction methods that transform verbose examples into compact, interpretable representations using structured classification criteria and explicit task descriptions. Through comprehensive experiments across five LLMs and varying few-shot ranges, we achieve up to 7\% macro-averaged AUC improvements over traditional few-shot in-context learning with up to 99\% reduction in token usage.

The interpretable nature of our generated artifacts enables effective user-in-the-loop collaboration, where users with varying expertise levels can directly modify classification logic. These contributions offer a practical solution for long-context classification tasks, addressing critical needs for transparency, efficiency, and user interaction in real-world applications.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: LLM Efficiency,few-shot learning,human-in-the-loop,business NLP,NLP datasets

Contribution Types: Model analysis & interpretability, Approaches to low-resource settings, Data resources

Languages Studied: English

Submission Number: 91