AppealCase: A Dataset and Benchmark for Civil Case Appeal Scenarios

Yuting Huang, Meitong Guo, Yiquan Wu, Ang Li, Mengze Li, Xiaozhong Liu, Keting Yin, Changlong Sun, Kun Kuang

Keywords: LegalAI, NLP, dataset

Paper Type: Full papers

Abstract: Recent advances in Legal Artificial Intelligence (LegalAI) have focused on single-case judgment analysis while largely overlooking the appellate process. Appeals serve as a vital mechanism for error correction and fair trials, making them crucial in both legal practice and AI research. The appellate scenario presents unique challenges for LLMs, including cross-trial factual dependencies, longer input contexts, and more fine-grained, complex legal reasoning. To address this gap, we introduce AppealCase, a dataset of 10,000 real-world pairs of matched first-instance and second-instance documents across 91 civil categories. AppealCase provides a dedicated annotation scheme along five key dimensions: judgment reversals, reversal reasons, cited legal provisions, claim-level decisions, and whether new information appears in the second instance. Based on these structured annotations, we define five benchmark tasks and evaluate 20 mainstream LLMs. Results show that current models struggle in the appellate setting—on Judgment Reversal Prediction, all models achieve F1 scores below 50%—highlighting the complexity and difficulty of appeal-focused LegalAI. We hope AppealCase fosters future work on appellate case understanding and contributes to more consistent judicial outcomes.

Submission Number: 40