Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMs

Carissa Cullen, Harry Garland, Alexander Roman, Louis Thomson, Christos Ziakas, Elliott Thornley

Keywords: shutdown problem, shutdownable agents, corrigibility, reinforcement learning, reward design

TL;DR: To test a proposed solution to the shutdown problem, we train agents to choose stochastically between different trajectory-lengths.

Abstract: Misaligned artificial agents might resist shutdown. One proposed solution is to train agents to lack preferences between different-length trajectories. The Discounted Reward for Same-Length Trajectories (DReST) reward function does this by penalizing agents for repeatedly choosing same-length trajectories, thus incentivizing agents to (1) choose stochastically between different trajectory-lengths (be NEUTRAL about trajectory-lengths), and (2) pursue goals effectively conditional on each trajectory-length (be USEFUL). In this paper, we use a DReST reward function to train deep RL agents and fine-tune LLMs to be NEUTRAL and USEFUL. We find that these DReST agents generalize to being NEUTRAL and USEFUL in unseen contexts at test time. Indeed, DReST RL agents achieve 11% (PPO) and 18% (A2C) higher USEFULNESS on our test set than agents trained with a more conventional reward function, and our fine-tuned LLM achieves maximum USEFULNESS and near-maximum NEUTRALITY. Our results provide some early evidence that DReST reward functions could be used to train more advanced agents to be USEFUL and NEUTRAL. Prior theoretical work suggests that these agents would be useful and shutdownable.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 48