A Summary of the COLIEE 2019 Competition

Juliano Rabelo, Mi-Young Kim, Randy Goebel, Masaharu Yoshioka, Yoshinobu Kano, Ken Satoh

Abstract: We summarize the evaluation of the 6th Competition on Legal Information Extraction/Entailment (COLIEE 2019). The competition consists of four tasks: two on case law and two on statute law. The case law component includes an information retrieval task (Task 1), and the confirmation of an entailment relation between an existing case and an unseen case (Task 2). The statute law component also includes an information retrieval task (Task 3) and an entailment/question answering task (Task 4), which attempts to confirm whether a particular statute applies to a yes/no question. Participation was open to any group in the world, based on any approach. Eleven different teams participated in the case law competition tasks, some of them in more than one task. We received results from 7 teams for Task 1 (15 runs) and 7 teams for Task 2 (18 runs). For the statute law tasks, 8 different teams participated, some in more than one task. Seven teams submitted a total of 13 runs for Task 3, and 7 teams submitted a total of 15 runs for Task 4. Here we summarize each team’s approaches, our official evaluation, and some analysis of the variety of methods that produced the evaluation results.