ClauseQA: Enhancing Customized Clause Extraction in Large Language Models via Instruction Following
Abstract: Contract review is a critical and time-consuming task for lawyers, involving the identification of key clauses that may pose potential risks.
However, previous methods trained on predefined taxonomies struggle to generalize to meet varying requirements.
To address this limitation, we propose ClauseQA, a framework to adapt large language models (LLMs) to extract clauses by following instructions with customized clause descriptions.
Additionally, we introduce an out-of-distribution setting for recognizing unseen clause categories, investigating how supervised fine-tuning (SFT) affects LLMs' generalization.
Our experiments show that SFT significantly reduces hallucinations while making LLMs more cautious in providing positive answers, which can sometimes lead to lower recall.
Furthermore, we observe that SFT tends to induce the original pre-training capability in decoder-only models like Llama3, whereas encoder-decoder models, such as Flan-T5, fit the SFT data more closely and thus show less robustness to distribution shifts.
Finally, we discuss potential directions for future research. Our code and models will be released.
Paper Type: Short
Research Area: NLP Applications
Research Area Keywords: legal NLP
Contribution Types: Model analysis & interpretability, Publicly available software and/or pre-trained models
Languages Studied: English
Submission Number: 2723
Loading