Data geometry and topology dependent bounds on network widths in deep ReLU networks
Primary Area: learning theory
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Dataset geometry, dataset topology, deep ReLU network, width bounds
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
TL;DR: We provide bounds on ReLU network widths, depends on geometric and topological characteristics of the dataset.
Abstract: The geometrical perspective of deep ReLU networks is important to understand the learning behavior and generalization capability of the neural networks. As such, here we investigate the relationship between the geometric and topological attributes of datasets and ReLU network architectures. Specifically, we first establish the data geometry-dependent bounds of the ReLU network widths and unveil a profound connection between these bounds and the underlying data manifold. Then, we show that topological characteristics are not the sole factor in fully determining network architecture. Rather, by combining the constraints on the hole shapes of the data manifold, the network architecture can be characterized by the Betti numbers of the data manifold. We further provide theoretical and empirical evidences that gradient descent converges to the proposed network configurations.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 2264
Loading