Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifolds

ICLR 2026 Conference Submission14936 Authors

Keywords: Fisher information, informaton geometry, Hutchinson's trick, deep learning theory

TL;DR: We discover an efficient way to estimate the Fisher information in deep classifiers with bounded variance.

Abstract: The high dimensional parameter space of modern deep neural networks — the neuromanifold — is endowed with a unique metric tensor defined by the Fisher information, estimating which is crucial for both theory and practical methods in deep learning. To analyze this tensor for classification networks, we return to a low dimensional space of probability distributions — the core space — and carefully analyze the spectrum of its Riemannian metric. We extend our discoveries there into deterministic bounds of the metric tensor on the neuromanifold. We introduce an unbiased random estimate of the metric tensor and its bounds based on Hutchinson’s trace estimator. It can be evaluated efficiently through a single backward pass, with a standard deviation bounded by the true value up to scaling.

Primary Area: learning theory

Submission Number: 14936