Benchmarking LLMs on the Semantic Overlap Summarization Task
Abstract: Semantic Overlap Summarization (SOS) is a constrained multi-document summarization task, where the constraint is to capture the common/overlapping information between two alternative narratives. While recent advancements in Large Language Models (LLMs) have achieved exceptional performance in numerous summarization tasks, a benchmarking study of the SOS task using LLMs is yet to be performed. As LLMs' responses are highly sensitive to variations in prompt design, a major challenge in conducting such a benchmarking study is to systematically explore a variety of prompts before drawing a reliable conclusion. Fortunately, the TELeR taxonomy has been recently proposed, which can be used to design and explore various prompts for LLMs. Using this TELeR taxonomy, this paper comprehensively evaluates 16 popular LLMs on the SOS Task.
Paper Type: Long
Research Area: Summarization
Research Area Keywords: Generation, Language Modeling, NLP Applications, Resources and Evaluation, Summarization
Contribution Types: Data resources, Data analysis
Languages Studied: English
Submission Number: 2882
Loading