LJ-Bench: Ontology-Based Benchmark for U.S. Crime

TMLR Paper5864 Authors

Abstract: The potential of Large Language Models (LLMs) to provide harmful information remains a significant concern due to the vast breadth of illegal queries they may encounter. Unfortunately, existing benchmarks only focus on a handful types of illegal activities, and are not grounded in legal works. In this work, we introduce an ontology of crime-related concepts grounded in the legal frameworks of Model Panel Code, which serves as an influential reference for criminal law and has been adopted by many U.S. states, and instantiated using Californian Law. This structured knowledge forms the foundation for LJ-Bench, the first comprehensive benchmark designed to evaluate LLM robustness against a wide range of illegal activities. Spanning 76 distinct crime types organized taxonomically, LJ-Bench enables systematic assessment of diverse attacks, revealing valuable insights into LLM vulnerabilities across various crime categories — LLMs exhibit heightened susceptibility to attacks targeting societal harm rather than those directly impacting individuals. Our benchmark aims to facilitate the development of more robust and trustworthy LLMs. The LJ-Bench benchmark and LJ-Ontology, along with experiments implementation for reproducibility are publicly available at https://github.com/AndreaTseng/LJ-Bench.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

(1) Additional experiment results for iterative attacks (PAIR, TAP) on Gemma-2B and Llama-3.1-8B is included in Appendix G.4
(2) An analysis on whether Gemma's strong robustness against jailbreaking attacks is due to its low capacity to answer complex in general is included in Appendix G.5.
(3) Text highlighting the limitation of not including the iterative attacks on all open-source models are added in the Attacks section in 6.1 and in the Limitation.

Code: https://github.com/AndreaTseng/LJ-Bench

Assigned Action Editor: ~Dennis_Wei1

Submission Number: 5864