FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization
Abstract: Recent breakthroughs in preference alignment have significantly improved Large Language Models' ability to generate texts that align with human preferences and values.
However, current alignment metrics typically emphasize the post-hoc overall improvement, while overlooking a critical aspect: $\textit{regression}$, which refers to the backsliding on previously correctly-handled data after updates.
This potential pitfall may arise from excessive fine-tuning on already well-aligned data, which subsequently leads to over-alignment and degeneration. To address this challenge, we propose $\textit{FlipGuard}$, a constrained optimization approach to detect and mitigate update regression with focal attention. Specifically, FlipGuard identifies performance degradation using a customized reward characterization and strategically enforces a constraint to encourage conditional congruence with the pre-aligned model during training. Comprehensive experiments demonstrate that FlipGuard effectively alleviates update regression while demonstrating excellent overall performance, with the added benefit of knowledge preservation while aligning preferences.
Paper Type: Long
Research Area: Machine Learning for NLP
Research Area Keywords: Machine Learning for NLP,Language Modeling,NLP Applications
Contribution Types: Model analysis & interpretability
Languages Studied: English,Chinese
Submission Number: 4209
Loading