Exploring Prompting Approaches in Legal Textual Entailment

Onur Bilgin, Logan Fields, Antonio Laverghetta, Zaid Marji, Animesh Nighojkar, Stephen Steinle, John Licato

Abstract: We report explorations into prompt engineering with large pre-trained language models that were not fine-tuned to solve the legal entailment task (Task 4) of the 2023 COLIEE competition. Our most successful strategy used simple text similarity measures to retrieve articles and queries from the training set. We report on our efforts to optimize performance with both OpenAI’s GPT-4 and FLaN-T5. We also used an ensemble approach to find the best combination of models and prompts. Finally, we analyze our results and suggest ideas for future improvements.