Dynamic Compression Strategies for Uniform Low-Dimensional Representations in Human Brain and Neural Network
Keywords: Low-Dimensional Representations, Generalization, Neural Manifold
Abstract: Recent studies suggest that the generalization performance of neural networks is strongly linked to their ability to learn low-dimensional data representations. However, limited attention has been given to the consistency of compression across different types of input data. In this work, we compute the intrinsic dimensions of raw data and their corresponding representations to quantify the extent of information compression in neural networks. Our results indicate that the pre-trained model CLIP compresses complex datasets significantly more than simpler ones and tends to represent diverse datasets with uniform low-dimensional manifolds. Similarly, we observe stable dimensionality in neural manifolds in the brain across various tasks and cognitive processes, suggesting that biological systems also favor consistent low-dimensional representations. Theoretically, we demonstrate that lower-dimensional manifolds increase the probability of interpolation, facilitating the representation of new samples as convex combinations of existing data. Additionally, we derive an upper bound on generalization error within the interpolation regime, which tightens as the dimensionality of the data decreases. These findings underscore the critical role of uniform low-dimensional manifolds in supporting efficient and generalizable information representation in both artificial and biological neural systems.
Primary Area: learning theory
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 7379
Loading