Causally Robust Preference Learning with Reasons

ICLR 2026 Conference Submission21131 Authors

Keywords: Preference-based learning, causal confusion, learning from human feedback

TL;DR: We develop a framework that utilizes natural language rationales to prevent causal confusion in preference learning.

Abstract:

Preference‑based reward learning is widely used for shaping agent behavior to match a user's preference, yet its sparse binary feedback makes it especially vulnerable to causal confusion. The learned reward often latches onto spurious features that merely co‑occur with preferred trajectories during training, collapsing when those correlations disappear or reverse at test time. We introduce ReCouPLe, a lightweight framework that uses natural language rationales to provide the missing causal signal. Each rationale is treated as a guiding projection axis in embedding space, training the model to score trajectories based on features aligned with that axis while de-emphasizing context that is unrelated to the stated reason. Because identical rationales can arise across multiple tasks (e.g., "_it avoids collisions with a fragile object_"), ReCouPLe naturally reuses the same causal direction whenever tasks share semantics, and transfers preference knowledge to novel tasks without extra data or language‑model fine‑tuning.
Our learned reward model can ground preferences on the articulated reason, aligning better with user intent and generalizing beyond spurious features.

Primary Area: causal reasoning

Submission Number: 21131