Guided Flow Policy: Workshop Version

Franki NGUIMATSIA TIOFACK, Théotime Le Hellard, Fabian Schramm, Nicolas Perrin-Gilbert, Justin Carpentier

Keywords: Reinforcement Learning, Flow Matching, Penalty Method

Abstract:

Offline reinforcement learning often relies on constraint optimization techniques that enforce policy improvement to remain close to the behavior policy. However, such approaches fail to distinguish between high-value and low-value actions in their penalty components.
We introduce Guided Flow Policy (GFP), which couples a multi-step flow-matching policy with a distilled one-step actor. The actor directs the flow policy through weighted behavior cloning to focus on cloning high-value actions from the dataset rather than indiscriminately imitating all state-action pairs. In turn, the flow policy constrains the actor to remain aligned with the dataset's best transitions while maximizing the critic. This mutual guidance enables GFP to achieve state-of-the-art performance across 144 tasks from the OGBench, Minari, and D4RL benchmarks, with substantial gains on suboptimal datasets and challenging tasks.

Submission Number: 42