Unintended Harmful Knowledge Elicitation Issue in Large Reasoning Models and a RL Solution

ICLR 2026 Conference Submission12948 Authors

Keywords: large reasoning model, reasoning safety, reinforcement learning

TL;DR: This paper tackles the issue of LRMs revealing harmful info unintentionally, and introduces a method to make them safer and more helpful.

Abstract: The capabilities of large language models (LLMs), particularly large reasoning models (LRMs), are rapidly advancing. This raises concerns about whether LRMs can maintain their safety awareness throughout long-form reasoning. Frustratingly, we identify a prevalent safety issue across LLMs and LRMs, where LRMs can reveal dangerous thoughts, leading to harmful knowledge elicitation when confronting sensitive yet benign topics. For example, when explaining the chemical context of Lewisite, a biological weapon, LRMs analyze its synthesis in their reasoning without recognizing the associated risks. We refer to this issue as the $\textit{unintended elicitation}$ issue. Experiments on our benchmark show that it is a common issue across current LRMs due to their strong multi-step reasoning capabilities. To address this issue, we propose placing LLMs in our synthesized open-ended environments, allowing them to self-search for a safety reasoning pattern to respond responsibly and helpfully. We first design a scalable data synthesis pipeline to generate data that triggers the ``$\textit{unintended elicitation}$'' issue. We further propose a safety-first reward model design, which prioritizes safety while also evaluating the helpfulness of responses and the faithfulness of reasoning. Experiments show that our method improves safety, reduces over-refusal, and maintains strong helpfulness, paving the way for safer deployment in high-stakes domains.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 12948