DISCOVERING AND EXPLAINING THE REPRESENTATION BOTTLENECK OF DNNS

Huiqi Deng, Qihan Ren, Hao Zhang, Quanshi Zhang

Keywords: representation bottleneck, representation ability, interaction, explanation

Abstract: This paper explores the bottleneck of feature representations of deep neural networks (DNNs), from the perspective of the complexity of interactions between input variables encoded in DNNs. To this end, we focus on the multi-order interaction between input variables, where the order represents the complexity of interactions. We discover that a DNN is more likely to encode both too simple and too complex interactions, but usually fails to learn interactions of intermediate complexity. Such a phenomenon is widely shared by different DNNs for different tasks. This phenomenon indicates a cognition gap between DNNs and humans, and we call it a representation bottleneck. We theoretically prove the underlying reason for the representation bottleneck. Furthermore, we propose losses to encourage/penalize the learning of interactions of specific complexities, and analyze the representation capacities of interactions of different complexities. The code is available at https://github.com/Nebularaid2000/bottleneck.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2111.06236/code)