Rule Based Rewards for Fine-Grained LLM Safety

Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian D Kivlichan, Molly Lin, Alex Beutel, John Schulman, Lilian Weng

Keywords: Large Language Model, LLM, RLHF, RLAIF, Safety, RBR, refusal, alignment

TL;DR: We propose Safety Rule Based Rewards, a method of using AI feedback in the safety training of LLMs that is scalable, allows for fine-grained control of behavior, and allows for fast updates.

Abstract: Reinforcement learning based fine-tuning of large language models (LLMs) on human preferences has been shown to enhance both their capabilities and safety behavior. However, in cases related to safety, without precise instructions to human annotators, the data collected may cause the model to become overly cautious, or to respond in an undesirable style, such as being judgmental. Additionally, as model capabilities and usage patterns evolve, there may be a need to add or relabel data to modify safety behavior. We propose a novel preference modeling approach that requires minimal human data and utilizes AI feedback. Our method, Rule Based Rewards (RBR), uses a collection of rules for desired or undesired behaviors (e.g. "refusals should not be judgmental") along with a LLM grader. In contrast to prior methods using AI feedback, our method uses fine-grained, composable, LLM-graded few-shot prompts as reward directly in RL training, resulting in greater control, accuracy and ease of updating. We show that RBRs are an effective training method, resulting in higher accuracy in safety-related performance compared to a human-feedback baseline.

Submission Number: 88