On Equivalences between Weight and Function-Space Langevin Dynamics

Ziyu Wang, Yuhao Zhou, Ruqi Zhang, Jun Zhu

Keywords: approximate inference, bayesian neural network

TL;DR: Weight-space Langevin dynamics does function-space inference.

Abstract: Approximate inference for overparameterized Bayesian models appears challenging, due to the complex structure of the posterior. To address this issue, a recent line of work has investigated the possibility of directly conducting approximate inference in "function space", the space of prediction functions. This note provides an alternative perspective to this problem, by showing that for many models -- including a simplified neural network model -- Langevin dynamics in the overparameterized "weight space" induces equivalent function-space trajectories to certain Langevin dynamics procedures in function space. Thus, the former can already be viewed as a function-space inference algorithm, with its convergence unaffected by overparameterization. We provide simulations on Bayesian neural network models, and discuss the implication of the results.