ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding

Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, Omer Levy

Submission Type: Regular Long Paper

Submission Track: Resources and Evaluation

Keywords: Long lexts, summarization, question answering, benchmark, zero-shot

TL;DR: We introduce ZeroSCROLLS, a zero-shot benchmark for natural language understanding over long texts, and conduct a comprehensive evaluation of both open-source and closed large language models on it.

Abstract:

We introduce ZeroSCROLLS, a zero-shot benchmark for natural language understanding over long texts, which contains only test and small validation sets, without training data. We adapt six tasks from the SCROLLS benchmark, and add four new datasets, including two novel information fusing tasks, such as aggregating the percentage of positive reviews. Using ZeroSCROLLS, we conduct a comprehensive evaluation of both open-source and closed large language models, finding that Claude outperforms ChatGPT, and that GPT-4 achieves the
highest average score. However, there is still room for improvement on multiple open challenges in ZeroSCROLLS, such as aggregation
tasks, where models struggle to pass the naive baseline. As the state of the art is a moving target, we invite researchers to evaluate their
ideas on the live ZeroSCROLLS leaderboard.

Submission Number: 406