LJ-Bench: Ontology-based Benchmark for Crime
Keywords: Ontology, Knowledge Graph, Crime, Language Models
Abstract: Despite the remarkable capabilities of Large Language Models (LLMs), their potential to provide harmful information remains a significant concern due to the vast breadth of illegal queries they may encounter. In this work, we firstly introduce structured knowledge in the form of an ontology of crime-related concepts, grounded in the legal frameworks of Californian Law and Model Penal Code. This ontology serves as the foundation for the creation of a comprehensive benchmark, called LJ-Bench, the first extensive dataset designed to rigorously evaluate the robustness of LLMs against a wide range of illegal activities. LJ-Bench includes 76 distinct types of crime, organized into a taxonomy. By systematically assessing the performance of diverse attacks on our benchmark, we gain valuable insights into the vulnerabilities of LLMs across various crime categories, indicating that LLMs exhibit heightened susceptibility to attacks targeting societal harm rather than those directly impacting individuals. Our benchmark aims to facilitate the development of more robust and trustworthy LLMs.
Primary Area: datasets and benchmarks
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3930
Loading