Bridging Law and Data: Augmenting Reasoning via a Semi-Structured Dataset with IRAC methodology

Anonymous

Abstract: Despite the advancements of Large Language Models (LLMs), their effectiveness in legal reasoning is limited due to unique legal terminologies and the need for highly specialized knowledge. These limitations can be addressed with high-quality data for complex legal reasoning. To this end, this paper introduces a benchmark, LegalSemi, annotated with IRAC (Issue, Rule, Application, Conclusion) for legal scenario analysis, developed by legal experts. It includes 54 legal scenarios annotated with full IRAC analysis and an associated structured knowledge graph (SKG). Our analysis reveals that Mistral-7b, a state-of-the-art LLM, is particularly adept at identifying legal concepts, while GPT-3.5 shows superior performance in analysis and conclusion tasks. Notably, standard LLMs face challenges in rule retrieval, an issue significantly mitigated by integrating SKG, which enhances the accuracy by 48%. LegalSemi serves as an innovative and valuable benchmark for complex legal reasoning, with the potential for broader applications across various legal domains.

Paper Type: long

Research Area: Resources and Evaluation

Contribution Types: Data resources

Languages Studied: English