Investigating Learning in Deep Neural Networks Using Layer-Wise Weight Change

Ayush Manish Agrawal, Atharva Tendle, Harshvardhan Sikka, Sahib Singh, Amr Kayid

Abstract: Understanding the learning dynamics of deep neural networks is of significant interest to the research community as it can provide insights into the black box nature of neural nets. In this work, we conduct a study which analyzes layer-wise learning trends by measuring the relative change in the weights of a Deep Neural Net during training. Through our controlled yet exhaustive set of experiments we were able to identify key trends which could lead to better understanding of how neural networks learn and make way for better training regimes. In our work we explore the learning trends in ubiquitous convolutional neural networks and datasets. Our work provides a simple yet novel approach to interpreting neural networks and is different from previous investigative studies.