Sample Average Approximation for Black-Box VI

Javier Burroni, Justin Domke, Daniel Sheldon

Keywords: Black-box VI, sample average approximation, quasi-Newton.

TL;DR: Our novel Black-Box VI approach simplifies optimization using SAA, quasi-Newton methods, and automated hyperparameters.

Abstract:

We present a novel approach for black-box VI that bypasses the difficulties of stochastic gradient ascent, including the task of selecting step-sizes.
Our approach involves using a sequence of sample average approximation (SAA) problems.
SAA approximates the solution of stochastic optimization problems by transforming them into deterministic ones.
We use quasi-Newton methods and line search to solve each deterministic optimization problem and present a heuristic policy to automate hyperparameter selection.
Our experiments show that our method simplifies the VI problem and achieves faster performance than existing methods.