Understanding Deep Neural Networks as Dynamical Systems: Insights into Training and Fine-tuning

Shufan Shen, Zhengqi Pei, Shuhui Wang, Qingming Huang

Keywords: Interpretation, Dynamical system, Expressive ability

Abstract: This paper offers an interpretation mechanism for understanding deep neural networks and their learning processes from a dynamical perspective. The aim is to uncover the relationship between the representational capacity of neural networks and the dynamical properties of their corresponding dynamical systems. To this end, we first interpret neural networks as dynamical systems by representing neural weight values as relationships among neuronal dynamics. Then, we model both neural network training and inference as the dynamical phenomena occurring within these systems. Built upon this framework, we introduce the concept of dynamical discrepancy, a macroscopic attribute that describes the dynamical states of neurons. Taking the generalization capability of neural models as a starting point, we launch a hypothesis: the dynamical discrepancy of neuromorphic-dynamical systems correlates with the representational capacity of neural models. We conduct dynamics-based conversions on neural structures such as ResNet, ViT, and LLaMA to investigate this hypothesis on MNIST, ImageNet, SQuAD, and IMDB. The experimental fact reveals that the relationship between these neural models' dynamical discrepancy and representational capacity aligns perfectly with our theoretical conjecture. Building upon these findings, we introduce a universal analytical approach tailored for neural models.

Primary Area: visualization or interpretation of learned representations

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5381