TropEx: An Algorithm for Extracting Linear Terms in Deep Neural Networks

Martin Trimmel, Henning Petzka, Cristian Sminchisescu

Keywords: linear regions, linear terms, deep learning theory, deep neural networks, rectified linear unit, relu network, piecewise linear function, tropical function

Abstract: Deep neural networks with rectified linear (ReLU) activations are piecewise linear functions, where hyperplanes partition the input space into an astronomically high number of linear regions. Previous work focused on counting linear regions to measure the network's expressive power and on analyzing geometric properties of the hyperplane configurations. In contrast, we aim to understand the impact of the linear terms on network performance, by examining the information encoded in their coefficients. To this end, we derive TropEx, a nontrivial tropical algebra-inspired algorithm to systematically extract linear terms based on data. Applied to convolutional and fully-connected networks, our algorithm uncovers significant differences in how the different networks utilize linear regions for generalization. This underlines the importance of systematic linear term exploration, to better understand generalization in neural networks trained with complex data sets.

One-sentence Summary: We propose an algorithm for extracting linear terms of piecewise linear deep neural network functions and apply it to study differences between convolutional and fully-connected networks.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Data: [CIFAR-10](https://paperswithcode.com/dataset/cifar-10)