Faithfulness and Content Selection in Long-Input Multi-Document Summarisation of U.S. Civil Rights Litigation
Abstract: Automatic summarisation of legal cases would reduce the burden on legal professionals and increase the accessibility of the law. However, the abstractive methods which dominate recent research are prone to hallucination. Despite the fact that this is a barrier to practical use, preventing hallucination is currently an understudied area in the legal domain. We conduct the first study at the intersection of legal, multi-document, and faithful summarisation. In particular, by introducing a BERT-based content selection mechanism, we achieve an improvement of 0.2614 in the probability of a generated summary being entailed by its source text compared to a naïve content selection baseline, and observe qualitative improvements. Further, we demonstrate possible improvements of 5.56 ROUGE-1 F1, 5.46 ROUGE-2 F1, 2.7 ROUGE-L F1, and 2.15 BERTScore over the state-of-the-art if a perfectly predictive classifier was used, demonstrating the importance of content selection for summary faithfulness and quality for long-input legal abstractive summarisation.
Paper Type: Long
Research Area: Summarization
Research Area Keywords: multi-document summarization, abstractive summarisation
Contribution Types: NLP engineering experiment
Languages Studied: english
Submission Number: 1636
Loading