The Effects of Reward Misspecification: Mapping and Mitigating Misaligned ModelsDownload PDF

29 Sept 2021, 00:35 (edited 14 Feb 2022)ICLR 2022 PosterReaders: Everyone
  • Keywords: reward misspecification, reinforcement learning, reward hacking, alignment, ml safety
  • Abstract: Reward hacking---where RL agents exploit gaps in misspecified proxy rewards---has been widely observed, but not yet systematically studied. To understand reward hacking, we construct four RL environments with different misspecified rewards. We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, and observation space noise. Typically, more capable agents are able to better exploit reward misspecifications, causing them to attain higher proxy reward and lower true reward. Moreover, we find instances of \emph{phase transitions}: capability thresholds at which the agent's behavior qualitatively shifts, leading to a sharp decrease in the true reward. Such phase transitions pose challenges to monitoring the safety of ML systems. To encourage further research on reward misspecification, address this, we propose an anomaly detection task for aberrant policies and offer several baseline detectors.
  • One-sentence Summary: We map out trends in reward misspecification and how to mitigate their impact.
  • Supplementary Material: zip
23 Replies