Automated Mapping of Legal Criteria to the Texts of Adjudicatory Decisions Using LLMs

Hannes Westermann, Vern R. Walker, Jaromir Savelka

Abstract: Legal reasoning, argumentation and decision making employ various structures, containing logically connected criteria when determining specific outcomes. Here, we examine whether large language models (LLMs) can automatically map decision texts to such criteria, by determining whether the decision maker found the criteria to be satisfied or not, and by providing explanations as to why the decision maker arrived at a particular decision. We introduce a Web-browser interface (CATLEX) to present the user with the LLM-generated determinations and explanations. We evaluate the ability of LLMs to generate such determinations and explanations, based on an experiment on 50 cases in the domain of military veterans’ disability appeals in the United States. A quantitative and qualitative analysis shows promising results, with the best model achieving an F1-score of 0.97 and explanations being generally accurate and useful. The results suggest new ways of reading decisions and developing legal arguments, with significant potential in, e.g., access to justice.

External IDs:doi:10.1145/3769126.3769236