When Are Bias-Free ReLU Networks Like Linear Networks?

Yedi Zhang, Andrew M Saxe, Peter E. Latham

Keywords: ReLU network, linear network, gradient flow, gradient descent

TL;DR: We show that two-layer bias-free ReLU networks cannot express nonlinear odd functions and have the same learning dynamics as linear networks under symmetry conditions on the dataset.

Abstract: We investigate the expressivity and learning dynamics of bias-free ReLU networks. We firstly show that two-layer bias-free ReLU networks have limited expressivity: the only odd function two-layer bias-free ReLU networks can express is a linear one. We then show that, under symmetry conditions on the data, these networks have the same learning dynamics as linear networks. This allows us to give closed-form time-course solutions to certain two-layer bias-free ReLU networks, which has not been done for nonlinear networks outside the lazy learning regime. While deep bias-free ReLU networks are more expressive than their two-layer counterparts, they still share a number of similarities with deep linear networks. These similarities enable us to leverage insights from linear networks, leading to a novel understanding of bias-free ReLU networks. Overall, our results show that some properties established for bias-free ReLU networks arise due to equivalence to linear networks, and suggest that including bias or considering asymmetric data are avenues to engage with nonlinear behaviors.

Student Paper: Yes

Submission Number: 11