RuleEdit: Benchmarking Rule-Level Knowledge Editing in Large Language Models

Jizhi Zhang, Dylan Xinming Hou, Zhaoyi Li, Muhammad Asif Ali, Gus Xia, Lijie Hu

Keywords: Knowledge Editing, Rule-level Editing

Abstract: Knowledge editing seeks to update language models without full retraining. While most prior work focuses on isolated factual or instance-level edits, we explore a more structured domain: mathematical rules. We introduce \textbf{RULE-EDIT}, the first benchmark explicitly designed for editing and evaluating rule-level abstract knowledge in LLMs. Beyond measuring direct edit accuracy, our benchmark is designed to encourage deeper investigation into the interpretability and symbolic reasoning capabilities of LLMs: (1) To what extent do edits to abstract rules propagate to derived instances? and (2) How well do token-level updates align with higher-level symbolic structures across formats? To evaluate this, we propose two new metrics:\emph{Instance Portability} and \emph{Rule Understanding} that quantify whether edits correctly generalize to rule-governed examples and maintain consistency across symbolic and natural language representations. Through experiments on best-performing open-source LLMs using representative editing methods, we find that while models can often overwrite formula-level knowledge, they frequently struggle to propagate these edits to rule-derived instances and to maintain consistency across different forms of a rule. For example, several methods achieve nearly 100\% reliability on direct rule queries, yet their rule-specific scores remain unsatisfactory (Instance Portability never exceeds 52\% and Rule Understanding stays below 26\%). Our findings highlight the limits of current editing methods and motivate rule editing as a testbed for controllable knowledge in LLMs.

Primary Area: datasets and benchmarks

Submission Number: 7455