BadReward: Clean-Label Poisoning of Reward Models in Text-to-Image RLHF

ICLR 2026 Conference Submission16599 Authors

Keywords: Data Poisoning Attack, Text-to-Image Models, Reinforcement Learning from Human Feedback

Abstract: Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning text-to-image (T2I) models with human preferences. However, RLHF's feedback mechanism also opens new pathways for adversaries. This paper demonstrates the feasibility of hijacking T2I models by poisoning a small fraction of preference training data with natural-appearing examples. Specifically, we propose BadReward, a stealthy clean-label poisoning attack targeting the reward model in T2I RLHF. BadReward operates by inducing feature collisions between visually contradicted preference data instances, thereby corrupting the reward model and subsequently compromising the T2I model's integrity. Unlike existing dirty-label alignment poisoning techniques focused on single (text) modality, BadReward is independent of the preference annotation process, enhancing its stealth and practical threat. Extensive experiments on popular T2I models show that BadReward can consistently guide the generation towards malicious outputs, such as biased or violent imagery, for targeted concepts. Our findings underscore the amplified threat landscape for RLHF in multi-modal systems, highlighting the urgent need for robust defenses.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 16599