RusConText Benchmark: A Russian Language Evaluation Benchmark for Understanding Context

Andrey Chirkin, Svetlana Kuznetsova, Maria Volina, Anna V. Dengina

Keywords: large language model, context understanding, benchmark

TL;DR: The paper presents new context understanding benchmark for Russian, encompassing ellipsis and coreference resolution, discourse and idiom understanding.

Abstract: This paper represents an implementation of an approach rather similar to that of (Zhu et al., 2024), but for the Russian-language data. We introduce a RusConText Benchmark for evaluating short context understanding in Russian, comprising four distinct yet interrelated tasks: ellipsis resolution, coreference resolution, and idiom interpretation, and discourse understanding. Each task targets a specific aspect of linguistic processing, challenging large language model to recover omitted information, resolve referential dependencies, interpret idioms and discourse. RusConText Benchmark serves as an additional resource beyond standard benchmarks, designed to assess model performance from a specific perspective. In addition, we present the results of scoring 4 models on our benchmark.

Archival Status: Archival

Paper Length: Long Paper (up to 8 pages of content)

Submission Number: 343