CaseBench: A GraphQL Benchmark for Indian Legal Text Analytics

ACL ARR 2024 December Submission2384 Authors

Abstract: Despite the integration of Large Language Models (LLMs) into legal workflows, several fundamental challenges remain in Legal Text Analytics (LTA). Many downstream tasks, such as determining case similarity or drafting complex legal documents, involve reasoning over large and heterogeneous data sources. Current models often struggle with factual consistency, hallucinations, and handling large contexts that integrate structured and unstructured data. To address these challenges, we introduce CaseBench, a new benchmark and resource that uses GraphQL as a retrieval mechanism for multi-modal legal data, enabling complex queries over relational tables, knowledge graphs, and vector databases. CaseBench provides data samples, query templates, and evaluation tasks designed to test the ability of LLMs to leverage GraphQL-based retrieval-augmented generation in legal contexts.

Paper Type: Short

Research Area: Resources and Evaluation

Research Area Keywords: large language models, legal text analytics, text to graphql

Contribution Types: Data resources

Languages Studied: English

Submission Number: 2384