Multiway Information Interaction Measures Reveal Redundant Coding in Biological and Artificial Neural Networks

Qiang Li

Keywords: multiway information interactions, redundancy representation, total correlation, data processing inequality, synergistic

Abstract: The investigation of information interaction in complex biological and deep neural networks is a significant research problem in neuroscience, deep learning, and information theory. However, there is limited research assessing multiway information interactions in the brain and deep neural networks, particularly regarding how information changes in each layer of neural networks. In this study, we explore redundancy representation in biological and standard deep convolutional neural networks. Firstly, we demonstrate that multiway information interaction, it can be quantified using total correlation, which indeed take precedence over pairwise metrics. Secondly, we propose that the data processing inequality holds in both biological vision and deep neural networks, and it confirms parallel information processing in the visual brain and deep neural networks. Furthermore, we observe that redundant information dominates in the early layers of the visual brain, while synergistic features gradually emerge in both the deeper layers of the visual cortex and deep neural networks, suggesting that general convolutional neural nets exhibit behavior similar to sensory information encoding in the primate visual brain.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 8196