Re-contextualization Mitigates Specification Gaming without Modifying the Specification

ICLR 2026 Conference Submission21139 Authors

Keywords: reinforcement learning, scalable oversight

Abstract: Developers often struggle to specify correct training labels and rewards. Perhaps they don't need to. We propose recontextualization, which reduces how often language models "game" training signals, performing misbehaviors those signals fail to penalize. We show recontextualization prevents models from learning to 1) prioritize evaluation metrics over chat response quality; 2) special-case code to pass incorrect tests; and 3) lie to users. Our method works by generating completions from prompts discouraging misbehavior and then recontextualizing them as though they were in response to prompts permitting misbehavior. Recontextualization trains language models to resist misbehavior even when instructions permit it. This mitigates the reinforcement of misbehavior from misspecified training signals, reducing specification gaming without improving the supervision signal.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 21139