Is it Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effort

ICLR 2026 Conference Submission13536 Authors

Keywords: Reward Hacking Detection, Chain-of-Thought Monitoring, Reasoning Faithfulness

TL;DR: TRACE detects implicit reward hacking by measuring how quickly truncated reasoning suffices to pass verification, outperforming CoT monitoring and enabling hidden loopholes discovery..

Abstract:

Reward hacking, where a reasoning model exploits loopholes in a reward function to achieve high rewards without solving the intended task, poses a significant threat.
This behavior may be explicit, i.e. verbalized in the model's chain-of-thought (CoT), or implicit, where the CoT appears benign thus bypasses CoT monitors.
To detect implicit reward hacking, we propose TRACE (Truncated Reasoning AUC Evaluation). Our key observation is that hacking occurs when exploiting the loophole is easier than solving the actual task.
This means that the model is using less `effort' than required to achieve high reward.
TRACE quantifies effort by measuring how early a model's reasoning becomes sufficient to pass a verifier.
We progressively truncate a model's CoT at various lengths and measure the verifier-passing rate at each cutoff. A hacking model, which takes a reasoning shortcut, will achieve a high passing rate with only a small fraction of its CoT, yielding a large area under the accuracy-vs-length curve.
TRACE achieves over 65% gains over our strongest 72B CoT monitoring baseline in math, and over 30% gains over a 32B monitoring baseline in code.
We further show that TRACE can discover unknown loopholes in the training environment.
Overall, TRACE offers a scalable unsupervised approach for oversight where current monitoring methods prove ineffective.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 13536