TMCL: Measuring Taiwanese Criminal Law Understanding

ACL ARR 2025 February Submission5261 Authors

Abstract: We present TMCL (Taiwan Multitask Criminal Law), a new benchmark designed for Taiwanese criminal legal questions in Traditional Chinese. TMCL provides 11 tasks with a total of 1,914 multi-choice questions, including Taiwan's official examinations and human-annotated datasets for basic and conceptual features. We evaluate 10 recent LLMs supporting Simplified Chinese or Traditional Chinese, and the results indicate that understanding Taiwan's criminal legal knowledge remains a challenging task. Our dataset is publicly available on: https://anonymous.4open.science/r/TMCL_ACL-8421/

Paper Type: Short

Research Area: NLP Applications

Research Area Keywords: legal NLP, automatic evaluation

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Reproduction study, Data resources

Languages Studied: Traditional Chinese

Submission Number: 5261