Bounds for the entropy and information of Gaussian scale mixtures with applications to Neural Networks

Lampros Gavalakis

Keywords: Probabilistic methods, Gaussian processes, Gaussian mixtures, Entropy, Fisher Information, Rates of Convergence

TL;DR: Convexity properties of information functionals and CLT rates for the Fisher information matrix of Gaussian scale mixtures.

Abstract:

We explore convexity properties of information functionals for
mixtures of centered Gaussian random variables (with respect
to the variance). First, we present a result concerning the concavity of entropy, which
confirms an open conjecture of Ball,
Nayar and Tkocz (2016) for this class of random variables.
Secondly, we show that the Fisher information matrix is operator-convex as a matrix-valued
function acting on densities of mixtures.
This result allows us to derive rates of convergence for the Fisher information matrix of the sum of weighted i.i.d. Gaussian mixtures in
the operator norm, along the Central Limit Theorem.
We explain how the distribution of the output of a Bayesian Neural Net with Gaussian priors admits a Gaussian mixture representation and, motivated by this, we discuss open questions related to extending the above rates to weakly dependent mixtures.

Based on joint work with Alexandros Eskenazis (CNRS, Sorbonne) and ongoing work with Ioannis Kontoyiannis (Cambridge).

Submission Number: 29