Post-Training Neural Network Pruning using Graph Curvature

TMLR Paper6972 Authors

Abstract:

This paper provides a fresh view of the neural network (NN) pruning problem, through the lens of graph theory. To achieve effective pruning, we aim to identify the main NN data flows and the corresponding NN connections that are most (and least) important for the performance of the full model.
Unlike the standard approach to NN data flow analysis, which is based on information theory, we employ the notion of graph curvature, specifically Ollivier-Ricci curvature (ORC). The ORC has been successfully used to identify important graph edges in various domains such as road traffic analysis, biological and social networks. In particular, edges with negative ORC are considered bottlenecks and as such are critical to the graph's overall connectivity, whereas positive-ORC edges are not essential. We use this intuition for the case of NNs as well: we 1) construct a graph induced by the NN structure and introduce the notion of neural curvature (NC) based on the ORC; 2) calculate curvatures based on activation patterns for a set of input examples; 3) aim to demonstrate that NC can indeed be used to rank edges according to their importance for the overall NN functionality.
We evaluate our method through pruning experiments on a variety of models trained on three image datasets, namely MNIST, CIFAR-10 and CIFAR-100.
The results indicate that our method can identify a larger number of unimportant edges as compared to existing pruning methods.

Submission Type: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

1. Change the first figure in Figure 13.
2. modified title, abstract
3. Changes in the paper are highlighted in red
4. Revised Figure 5 and 8, Section 6.3.2 and 6.3.5

Assigned Action Editor: ~Nadav_Cohen1

Submission Number: 6972