AMHR COLIEE 2024 Entry: Legal Entailment and Retrieval

Animesh Nighojkar, Kenneth Jiang, Logan Fields, Onur Bilgin, Stephen Steinle, Yernar Sadybekov, Zaid Marji, John Licato

Abstract: This paper presents the methodologies and results of the participation of the Advanced Machine Human Reasoning (AMHR) group in the 2024 Conference on Legal Information and Information Technology (COLIEE). Our team participated in Tasks 2, 3, and 4, which focused on the legal case entailment, the retrieval of statute laws, and the legal textual entailment, respectively. In Task 2, we explore two approaches to identify paragraphs from older cases relevant to a given case fragment. The first approach used a fine-tuned legalBERT model, which resulted in overfitting, while the second approach employed a fine-tuned monoT5 model, augmented with hard negative mining, which won the competition for Task 2. In Task 3, our strategy involved sorting Civil Code articles using a fine-tuned MonoT5 model, followed by a large language model with post-processing for article selection, prioritizing the F2-score. Task 4 involved various prompting strategies using Google’s flan-T5-xxl model, with a focus on ranked preference voting for top prompts to identify the entailment of legal queries from legal articles. Our methodologies used advanced deep learning techniques, tailored to the specific legal domain, and were supported by solid engineering practices, enabling competitive outcomes in the COLIEE 2024 competition. This paper details our approaches, providing information on the challenges and innovations in the field of legal AI.