Generating Clarifying Questions for Conversational Legal Case Retrieval without External Knowledge

Bulou Liu, Yiran Hu, Qingyao Ai, Yiqun Liu, Yueyue Wu, Chenliang Li, Weixing Shen

Abstract: In legal case retrieval, existing work has shown that human-mediated conversational search can improve users’ search experience. One of the key problems for a practical conversational search system is how to ask high-quality clarifying questions to initiate conversations with users and understand their search intents. Previous works demonstrated that human-annotated external domain knowledge (such as event schemas) can improve the legal utility of clarifying questions generated by large language models. However, these methods are restricted to specific law systems or languages and cannot be generalized to others. To this end, we propose to generate context and domain-specific questions with LLMs without external annotations or knowledge by extracting information from top-retrieved documents given the current conversation context. Specifically, we construct a conversational legal case retrieval system CARQ that iteratively selects neighbor candidate case documents from the retrieved list at each conversation step to ask clarifying questions. We pretrain CARQ to capture the differences between legal cases and employ the reward augmented maximum likelihood to optimize the system directly for retrieval metrics. Extensive automated and human evaluations on three widely adopted legal case retrieval datasets demonstrate the superior effectiveness of our approach as compared with the state-of-the-art baselines.