PyLegalIR: A Benchmark for Spanish Legal Information Retrieval from Paraguayan Supreme Court Cases

ACL ARR 2025 May Submission5101 Authors

Abstract: We present PyLegalIR, a benchmark dataset designed for evaluating information retrieval systems in the Spanish legal domain, using Criminal Chamber cases of the Paraguayan Supreme Court (SCP). Despite the critical need for effective legal retrieval systems in the Spanish Language, there are no publicly available datasets. PyLegalIR addresses this gap by providing a supervised benchmark comprising 54 expert-created queries, each annotated with 30 relevant documents on average, resulting in 1,597 query-document pairs with graded relevance judgments. Annotations were performed by Paraguayan legal professionals, covering diverse legal topics. This dataset enables benchmarking and fosters the development of retrieval systems for Spanish legal texts. All code and data are publicly available.

Paper Type: Short

Research Area: Information Retrieval and Text Mining

Research Area Keywords: dense retrieval, document representation, re-ranking, legal NLP, corpus creation, benchmarking, data augmentation, NLP in resource-constrained settings, transfer learning / domain adaptation, generalization, NLP datasets, reproducibility

Contribution Types: Approaches to low-resource settings, Data resources

Languages Studied: Spanish

Submission Number: 5101