Towards Understanding The Winner-Take-Most Behavior Of Neural Network Representations
Keywords: deep learning, neuron representations
TL;DR: Generalizing neural networks exhibit a remarkable winner-take-most phenomenon at the level of their neuron-level internal representations
Abstract: Understanding the generalization ability of neural networks is a long-standing goal of the machine learning community.
Despite sufficient representational complexity to memorize large data sets, modern neural networks are able to learn solutions to their optimization problem that generalize well to unseen data.
In this paper, we explore how the neuron-level representations of the training samples in a data set can be analyzed to differentiate between networks that have learned to generalize and networks that merely memorize.
For this purpose, we introduce a synthetic data set specifically crafted to allow for an easy comparison of how networks treat simple patterns.
We show that comparing how training samples presenting different patterns are represented by neurons can provide key insights as to what differentiates memorized and generalized networks.
We observe that the training process progressively increases the average pre-activation of the most activated patterns of a class and decreases the average pre-activation of the least activated patterns of said class in each neuron, a winner-take-most phenomenon.
In order to solve the classification problem, the network seems to apply a divide-and-conquer strategy, where different neurons specialize in the classification of different patterns of a class.
We also explore the effect of various parameters of our experimental configuration on these findings and describe three necessary conditions for it to appear.
Finally, we provide an intuitive explanation of why this phenomenon occurs, drawing links with existing work on sample difficulty, coherent gradients, and implicit clustering.
Primary Area: visualization or interpretation of learned representations
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 8583
Loading