Convolutional Neural Network Dynamics: A Graph Perspective

Fatemeh Vahedian, Ruiyu Li, Puja Trivedi, Di Jin, Danai Koutra

Keywords: neural network dynamics, time-evolving graphs, interpretation of neural networks, performance prediction

Abstract:

The success of neural networks (NNs) in a wide range of applications has led to increased interest in understanding the underlying learning dynamics of these models. In this paper, we go beyond mere descriptions of the learning dynamics by taking a graph perspective and investigating the relationship between the graph structure of NNs and their performance.
Specifically, we propose (1) representing the neural network learning process as a time-evolving graph (i.e., a series of static graph snapshots over epochs), (2) capturing the structural changes of the NN during the training phase in a simple temporal summary, and (3) leveraging the structural summary to predict the accuracy of the underlying NN in a classification or regression task. For the dynamic graph representation of NNs, we explore structural representations for fully-connected and convolutional layers, which are key components of powerful NN models. Our analysis shows that a simple summary of graph statistics, such as weighted degree and eigenvector centrality, over just a few epochs, can be used to accurately predict the performance of NNs. For example, a weighted degree-based summary of the time-evolving graph that is constructed based on 5 training epochs of the LeNet architecture achieves classification accuracy of over 93\%. Our findings are consistent for different NN architectures, including LeNet, VGG, AlexNet, and ResNet.

One-sentence Summary: This paper seeks to understand the learning process of NNs and predict their performance from a few epochs by taking a graph perspective: it represents the training process of NNs as a dynamic graph and summarizes it via simple graph statistics.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2111.05410/code)