The Contraction Property of Pooling Layer
Keywords: Average Pooling, Max Pooling, CNN, Deep Learning
Abstract: Although the theory of deep neural networks has been studied for years, the mechanism of pooling layers is still elusive. In this paper, we report the angle contraction behavior of pooling strategies (the average pooling and max pooling) at initialization. Compared to the relu-activated fully connected layer or convolutional layer, the pooling layer stands as the main source of contraction of the angle between hidden features. Moreover, we show that the cosine similarity between average pooling features in convolutional neural network is more data-dependent than fully connected network, while the max pooling is not sensitive to the data distribution in both architectures. Our results may complement the understanding of the representation learning.
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 10697
Loading