M$^4$LE: A Multi-Ability Multi-Range Long Context Evaluation Benchmark for Large Language Models
Primary Area: datasets and benchmarks
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: long-context, natural language understanding, large language model, evaluation, benchmark
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: Managing long sequences has become an important and necessary feature for large language models (LLMs). However, it is still an open question of how to comprehensively and systematically evaluate the long-sequence capability of LLMs. One of the reasons is that conventional and widely-used benchmarks mainly consist of short sequences. In this paper, we propose $\textbf{M$^4$LE}$, a $\textbf{M}$ulti-ability, $\textbf{M}$ulti-range, $\textbf{M}$ulti-task, $\textbf{M}$ulti-domain benchmark for $\textbf{L}$ong-context $\textbf{E}$valuation.
$\textbf{M$^4$LE}$ is based on a diverse NLP task pool comprising
36 NLP datasets, 12 task types and 12 domains.
To alleviate the scarcity of tasks with naturally long sequences and incorporate multiple-ability assessment, we propose an automatic approach (but with negligible human annotations) to convert short-sequence tasks into a unified long-sequence scenario where LLMs have to identify single or multiple relevant spans in long contexts based on explicit or semantic hints.
Specifically, the scenario includes five different types of abilities: (1) explicit single-span; (2) semantic single-span; (3) explicit multiple-span; (4) semantic multiple-span; and (5) global context understanding.
The resulting samples in $\textbf{M$^4$LE}$ are evenly distributed from 1k to 8k input length.
We conducted a systematic evaluation on 11 well-established LLMs, especially those optimized for long-sequence inputs. Our results reveal that: 1) Current LLMs struggle to understand long context, particularly when tasks require multiple-span attention. 2) Semantic retrieval task is more difficult for competent LLMs. 3) Models fine-tuned on longer text with position interpolation have comparable performance to those using Neural Tangent Kernel (NTK) aware scaling methods without fine-tuning.
We make our benchmark publicly available to encourage future research in this challenging area.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5186
Loading