ChatGPT Summarization: A Deep Dive into In-Context Learning Efficacy

Anonymous

Abstract: Large language models (LLMs), such as ChatGPT, have risen to prominence in text summarization tasks, primarily due to the advent of in-context learning. This paper delves into how in-context learning steers the outputs of LLMs based on different data demonstration configurations. Our pivotal findings reveal that ChatGPT's adaptability to target summarization tasks is enhanced when provided with paired text and summaries, compared to when provided in isolation. Furthermore, the structured presentation of these pairs proves more influential than their precise content alignment. However, there are observable limitations: increasing the number of demonstrations yields diminishing returns, and the improvement of adaptability declines when tasked with more intricate news texts as opposed to simpler dialogues. This study comprehensively explains in-context learning's nuances in text summarization, highlighting its merits and demerits for future researchers.

Paper Type: short

Research Area: Summarization

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Consent To Share Submission Details: On behalf of all authors, we agree to the terms above to share our submission details.