Measuring Model Robustness via Fisher Information: Spectral Bounds, Theoretical Guarantees, and Practical Algorithms

Chong Zhang, Xiang Li, Jia Wang, Qiufeng Wang, Xiaobo Jin

Keywords: Model Robustness, Fisher Information Matrix, Spectral Norm, Architectural Complexity, Jacobian Matrix

TL;DR: Through innovative Fisher information matrix spectral analysis, we demonstrate that when matched to the data characteristics, simple models can outperform complex architectures in terms of robustness.

Abstract: The robustness of deep neural networks is critical for their deployment in safety-sensitive domains. This paper establishes a novel theoretical framework for quantifying model robustness through the lens of Fisher information. We first start with the known conclusion that maximizing the KL divergence of the posterior probability is equivalent to minimizing half the Mahalanobis distance defined by the Fisher Information Matrix (FIM), and further reveal that the FIM is equal to the variance of the input Jacobian matrix. Based on this insight, we propose the FIM's principal eigenvalue (or its reciprocal) as a principled robustness metric. We derive closed-form spectral bounds for common architectural components (e.g., ReLU, convolution) and theoretically compare the robustness of VGG, ResNet, DenseNet, and Transformer. To enable scalable computation, we resort to efficient algorithms, including power iteration and randomized Hutchinson, to estimate the robustness metric. Furthermore, we propose to use Hutchinson and finite differences to achieve robust estimation in a black-box setting. Extensive experiments validate our theoretical claims and demonstrate the metric's utility in predicting adversarial vulnerability. Code: https://anonymous.4open.science/r/8F4D7E6R/

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 7090