Reason-KE++: Aligning the Process, Not Just the Outcome, for Faithful LLM Knowledge Editing

ACL ARR 2026 January Submission9129 Authors

Keywords: chain-of-thought, fine-tuning, prompting, multihop QA, reasoning, NLP in resource-constrained settings, data-efficient training, data augmentation, PPO, reinforcement learning

Abstract: Aligning Large Language Models (LLMs) to be faithful to new knowledge in complex, multi-hop reasoning tasks is a critical, yet unsolved, challenge. We find that SFT-based methods, e.g., Reason-KE, while state-of-the-art, suffer from a "faithfulness gap": they optimize for format mimicry rather than sound reasoning. This gap enables the LLM's powerful parametric priors to override new contextual facts, resulting in critical factual hallucinations (e.g., incorrectly reasoning "Houston" from "NASA" despite an explicit edit). To solve this core LLM alignment problem, we propose Reason-KE++, an SFT+RL framework that instills process-level faithfulness. Its core is a Stage-aware Reward mechanism that provides dense supervision for intermediate reasoning steps (e.g., Decomposition, Sub-answer Correctness). Crucially, we identify that naive outcome-only RL is a deceptive trap for LLM alignment: it collapses reasoning integrity (e.g., 19.00\% Hop acc) while superficially boosting final accuracy. Our process-aware framework sets a new SOTA of 95.48\% on MQUAKE-CF-3k (+5.28\%), demonstrating that for complex tasks, aligning the reasoning process is essential for building trustworthy LLMs.

Paper Type: Long

Research Area: Mathematical, Symbolic, Neurosymbolic, and Logical Reasoning

Research Area Keywords: logical reasoning, deductive reasoning, neurosymbolic reasoning, self-supervised learning, data augmentation, generalization, reinforcement learning,

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Approaches low compute settings-efficiency, Data resources

Languages Studied: English

Submission Number: 9129