SummIt: Iterative Text Summarization via ChatGPT

Haopeng Zhang, Xiao Liu, Jiawei Zhang

Submission Type: Regular Long Paper

Submission Track: Summarization

Submission Track 2: Theme Track: Large Language Models and the Future of NLP

Keywords: summarization, large language model, text editing

TL;DR: The paper proposes an iterative text summarization framework that enables the model to refine the generated summary iteratively through self-evaluation and feedback.

Abstract:

Existing text summarization systems have made significant progress in recent years, but typically generate summaries in a single step. The one-shot summarization setting is sometimes inadequate, however, as the generated summary may contain hallucinations or overlook important details related to the reader's interests. In this paper, we address this limitation by proposing SummIt, an iterative text summarization framework based on large language models like ChatGPT.
Our framework enables the model to refine the generated summary iteratively through self-evaluation and feedback, closely resembling the iterative process humans undertake when drafting and revising summaries. Furthermore, we explore the potential benefits of integrating knowledge and topic extractors into the framework to enhance summary faithfulness and controllability. We evaluate the performance of our framework on three benchmark summarization datasets through empirical and qualitative analyses. We also conduct a human evaluation to validate the effectiveness of the model's refinements and find a potential issue of over-correction.

Submission Number: 442