How Does RLHF Shift Behavior Distributions? Distinguishability and Steerability
Primary Area: societal considerations including fairness, safety, privacy
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: RLHF, Alignment, Distinguishability, Steerability
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: Large Language Models (LLMs) have shown impressive capabilities, but their potential for causing harm has raised concerns. This paper delves into the impact of a common alignment approach, Reinforcement Learning from Human Feedback (RLHF), on an LLM's susceptibility to having its behavior steered into negative territory under persona prompts. We provide a systematic study to understand RLHF's effects on behavior distributions and the resulting vulnerabilities to prompt steering. In particular, we conceptualize LLM outputs as a decomposition of behaviors into positive and negative sub-distributions. Based on the decomposition, we first examine how RLHF influences the distinguishability between these sub-distributions across a wide spectrum of behaviors. Subsequently, we investigate behavioral steerability by devising persona prompts of varying lengths for each behavior in consideration. Our findings reveal that the RLHF model can be steered to exhibit more negative behavior, resulting in a significantly higher misalignment rate compared to the base model. However, the extent of this susceptibility does not appear to be predicted by the degree of distinguishability observed in the behavior sub-distributions.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 1944
Loading