Fourier Sensitivity and Regularization of Computer Vision Models

Kiran Krishnamachari, See-Kiong Ng, Chuan-Sheng Foo

Abstract: Recent work has empirically shown that deep neural networks latch on to the Fourier statistics of training data and show increased sensitivity to Fourier-basis directions in the input. Understanding and modifying this Fourier-sensitivity of computer vision models may help improve their robustness, hence, in this paper we study the frequency sensitivity characteristics of deep neural networks using a principled approach. We first propose a $\textbf{\textit{basis trick}}$, proving that unitary transformations of the input-gradient of a function can be used to compute its gradient in the basis induced by the transformation. Using this result, we propose a general measure of any differentiable computer vision model's $\textit{\textbf{Fourier-sensitivity}}$ using the unitary Fourier-transform of its input-gradient. When applied to deep neural networks, we find that computer vision models are consistently sensitive to particular frequencies dependent on the dataset, training method and architecture. Based on this measure, we further propose a $\textit{\textbf{Fourier-regularization}}$ framework to modify the Fourier-sensitivities and frequency bias of models. Using our proposed regularizer-family, we demonstrate that deep neural networks obtain improved classification accuracy on robustness evaluations.

Submission Length: Regular submission (no more than 12 pages of main content)

Code: https://github.com/kiranchari/Fourier-Sensitivity-Regularization

Assigned Action Editor: ~Matthew_Blaschko1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 368