CoLE: A collaborative legal expert prompting framework for large language models in law

Bo Li, Shuang Fan, ShaoLin Zhu, Lijie Wen

Abstract: Large Language Models (LLMs) have achieved remarkable outcomes in various natural language processing tasks. However, their application to the highly specialized field of law presents unique challenges. Legal language, characterized by complex syntax, domain-specific terminology, and nuanced logical relationships, poses significant hurdles for existing NLP models in accurately understanding and processing legal queries. Furthermore, the sheer volume of legal documents complicates information retrieval and knowledge extraction, making it difficult for models to pinpoint relevant legal articles and cases. Moreover, existing legal LLMs often struggle to effectively handle colloquial user queries and lack efficient mechanisms for selecting the most relevant demonstrations in In-Context Learning (ICL), hindering their ability to provide accurate and comprehensive legal advice. In order to address these issues, we propose a novel prompting framework named “Collaborative Legal Experts” (CoLE). This framework draws inspiration from teamwork paradigms in real-world legal case processing. First, we design an intent identification module to analyze user queries for identifying potential intents and law domains. Then, through two subsequent processes, potential background information and the best demonstration are generated. Finally, we design a prompt generator to assemble prompts generated from the previous steps. It combines with the LLMs to generate the final answer. Notably, we find that the self-generated information by LLMs has a smaller gap when fused with LLMs. We evaluate performance by integrating it with 7 general-purpose Chinese LLMs and comparing its performance against 8 specialized legal LLMs across 10 datasets, including Single-Choice, Multiple-Choice, and Question&Answer. The results indicate that integrating with CoLE’s LLMs has the potential to significantly enhance performance in the law field, particularly without the need for annotated datasets or model parameter updates. Moreover, our proposed model outperforms all state-of-the-art LLMs in law. The code is available at https://github.com/liboaccn/cole.