Scale-Invariant Empirical-Bayes Laplace Approximation for ReLU Networks

Shivam Pal, Piyush Rai

Keywords: Laplace Approximation, ReLU Scale Symmetry

TL;DR: Standard empirical-Bayes Laplace depends not only on the learned ReLU predictor, but also on the arbitrary scaling of its weights. We make it scale-invariant

Abstract: A post-hoc Bayesian procedure should be invariant to function-preserving reparameterizations. Standard empirical-Bayes Laplace violates this principle in ReLU networks. Positive homogeneity creates scale-equivalent parameterizations, but the isotropic evidence objective depends on the chosen representative through the prior norm and curvature log-determinant. We introduce $\gamma$-canonicalization, which fixes a canonical representative on each ReLU scale orbit, fits an isotropic Laplace approximation there, and transports the prior back to the original coordinates. The resulting precision field transforms compatibly with curvature, making the generalized evidence and Jacobian-linearized predictive distribution invariant for every $\gamma\in[0,1]$. Selecting $\gamma$ and the prior precision jointly by marginal likelihood yields an EB-Laplace procedure defined on ReLU scale-equivalence classes. Empirically, standard EB-Laplace varies across functionally identical rescalings, while $\gamma$-canonicalized Laplace removes this variation to zero.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 72