David vs. Goliath: Can small models leverage LLMs for summarization?
Abstract: Recent studies indicate a preference for summaries generated using large language models (LLMs) over those using classical models, highlighting a performance discrepancy. This study explores strategies to narrow the gap between the summaries generated through these two models. To address this, we introduce a novel framework that uses LLM-generated summaries to train classical models, adopting a two-stage training approach to enhance their summary quality. Although classical models are relatively smaller in size than LLMs, through automatic metrics and human evaluations, we can demonstrate that the performances of classical models, trained using LLM-generated references can catch up with LLM. Our findings create a simple yet potential way to improve classical summarization models by leveraging LLMs. Additionally, we contribute a new dataset $\textbf{GXSum}$, enabling further research and promoting development progress in this subject.
Paper Type: Long
Research Area: Summarization
Research Area Keywords: abstractive summarisation, pre-training, fine-tuning, corpus creation, data influence
Contribution Types: Model analysis & interpretability, Data analysis, Position papers
Languages Studied: English
Submission Number: 151
Loading