Distributional GFlowNets with Quantile Flows

Dinghuai Zhang, Ling Pan, Ricky T. Q. Chen, Aaron Courville, Yoshua Bengio

Authors that are also TMLR Expert Reviewers: ~Yoshua_Bengio1

Abstract:

Generative Flow Networks (GFlowNets) are a new family of probabilistic samplers where an agent learns a stochastic policy for generating complex combinatorial structure through a series of decision-making steps.
There have been recent successes in applying GFlowNets to a number of practical domains where diversity of the solutions is crucial, while reinforcement learning aims to learn an optimal solution based on the given reward function only and fails to discover diverse and high-quality solutions.
However, the current GFlowNet framework is relatively limited in its applicability and cannot handle stochasticity in the reward function.
In this work, we adopt a distributional paradigm for GFlowNets, turning each flow function into a distribution, thus providing more informative learning signals during training.
By parameterizing each edge flow through their quantile functions, our proposed \textit{quantile matching} GFlowNet learning algorithm is able to learn a risk-sensitive policy, an essential component for handling scenarios with risk uncertainty.
Moreover, we find that the distributional approach can achieve substantial improvement on existing benchmarks compared to prior methods due to our enhanced training algorithm, even in settings with deterministic rewards.

Certifications: Expert Certification

Submission Length: Regular submission (no more than 12 pages of main content)

Code: https://github.com/zdhNarsil/Distributional-GFlowNets

Assigned Action Editor: ~Andriy_Mnih1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 1655