Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward NetworksDownload PDFOpen Website

2021 (modified: 14 Nov 2022)ICLR 2021Readers: Everyone
Abstract: In this paper, we apply harmonic distortion analysis to understand the effect of nonlinearities in the spectral domain. Each nonlinear layer creates higher-frequency harmonics, which we call "blueshift", whose magnitude increases with network depth, thereby increasing the “roughness” of the output landscape. Unlike differential models (such as vanishing gradients, sharpness), this provides a more global view of how network architectures behave across larger areas of their parameter domain. For example, the model predicts that residual connections are able to counter the effect by dampening corresponding higher frequency modes. We empirically verify the connection between blueshift and architectural choices, and provide evidence for a connection with trainability.
0 Replies

Loading