What it means by learning in a neural network: easing the knot

Atiya Ibnat Tasnim, Rushmila Shehreen Khan, Md. Shahriar Karim

Keywords: Learning Dynamics, Complexity-Entropy, Information Bottleneck

Abstract: What constitutes learning in humans and machines remains puzzling despite the unprecedented growth we have witnessed in both. Starting with a Perceptron and, in subsequent interrogation of multilayer perceptron (MLP) and deep linear neural network (DLN), this paper revisits and searches for new learning signatures and dynamics in artificial neural networks (ANN). Precisely, we consider the transport of the initial weight distribution to its final form while optimally balancing entropy (randomness in the weights) and statistical complexity, which captures the neural network's information storage structure. As found, training neural networks guided through complexity-entropy improves its reproducibility. In continuation, we further assess depth dependence and information flow using entropy-difference and KL-divergence of weight distribution between successive layers. Insights obtained so far in our ongoing analysis of perceptron learning are of immense importance, with applications ranging from explainable AI to understanding brain function.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 209