Bounds for the entropy and information of Gaussian scale mixtures with applications to Neural Networks

Published: 23 Jun 2025, Last Modified: 23 Jun 2025Greeks in AI 2025 PosterEveryoneRevisionsBibTeXCC BY 4.0
Keywords: Probabilistic methods, Gaussian processes, Gaussian mixtures, Entropy, Fisher Information, Rates of Convergence
TL;DR: Convexity properties of information functionals and CLT rates for the Fisher information matrix of Gaussian scale mixtures.
Abstract: We explore convexity properties of information functionals for mixtures of centered Gaussian random variables (with respect to the variance). First, we present a result concerning the concavity of entropy, which confirms an open conjecture of Ball, Nayar and Tkocz (2016) for this class of random variables. Secondly, we show that the Fisher information matrix is operator-convex as a matrix-valued function acting on densities of mixtures. This result allows us to derive rates of convergence for the Fisher information matrix of the sum of weighted i.i.d. Gaussian mixtures in the operator norm, along the Central Limit Theorem. We explain how the distribution of the output of a Bayesian Neural Net with Gaussian priors admits a Gaussian mixture representation and, motivated by this, we discuss open questions related to extending the above rates to weakly dependent mixtures. Based on joint work with Alexandros Eskenazis (CNRS, Sorbonne) and ongoing work with Ioannis Kontoyiannis (Cambridge).
Submission Number: 29
Loading