U-Statistics for Importance-Weighted Variational Inference

Javier Burroni, Kenta Takatsu, Justin Domke, Daniel Sheldon

Abstract:

We propose the use of U-statistics to reduce variance for gradient estimation in importance-weighted variational inference.
The key observation is that, given a base gradient estimator that requires $m > 1$ samples and a total of $n > m$ samples to be used for estimation, lower variance is achieved by averaging the base estimator on overlapping batches of size $m$ than disjoint batches, as currently done.
We use classical U-statistic theory to analyze the variance reduction, and propose novel approximations with theoretical guarantees to ensure computational efficiency.
We find empirically that U-statistic variance reduction can lead to modest to significant improvements in inference performance on a range of models, with little computational cost.

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission: Camera ready

Assigned Action Editor: ~Tom_Rainforth1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 544