Understanding ResNet from a Discrete Dynamical System Perspective

Lijuan Zhang

Keywords: the condition number of modules, median principal angle, network mathematical description

Abstract: Residual network (ResNet) is one of popular networks proposed in recently years. Discussion about its theoretical properties is helpful for the understanding of networks with convolution modules. In this paper, we formulate the learning process of ResNet as a iterative system, then we may apply tools in discrete dynamical systems to explain its stability and accuracy. Due to the backward propagation of learning process, the module operations vary with the change of different layers. So we introduce the condition number of modules to describe the perturbation of output data, which can demonstrate the robustness of ResNet. In addition, the inter-class and intra-class median principal angle is defined to analyze the classification efficiency of ResNet. Mathematical description of the learning process of ResNet is given in a modular manner so that our research framework can be applied to other networks. In order to verify the feasibility of our idea, several experiments are carried out on the Dogs vs. Cats dataset, Kaggle: Animals 10 dataset, and ImageNet 2012 dataset. Simulation results are accordance with the theoretical analysis and prove the validity of our theory.

One-sentence Summary: In this paper, we view ResNet as a discrete dynamical system and give a mathematical description of the stability and classification accuracy of ResNet using perturbation theory and the concept of median principal angle.