Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Condition

TMLR Paper6138 Authors

Abstract: We study the classical binary classification problem for hypothesis spaces of Deep Neural Networks (DNNs) with ReLU activation under Tsybakov's low-noise condition with exponent $q > 0$, as well as its limit case $q = \infty$, which we refer to as the \textit{hard margin condition}. We demonstrate that DNN solutions to the empirical risk minimization (ERM) problem with square loss surrogate and $\ell_p$ penalty on the weights $(0 < p < \infty)$ can achieve excess risk bounds of order $\mathcal{O}\left(n^{-\alpha}\right)$ for arbitrarily large $\alpha > 1$ under the hard-margin condition, provided that the Bayes regression function $\eta$ satisfies a \textit{distribution-adapted} smoothness condition relative to the marginal data distribution $\rho_X$. Additionally, we establish minimax lower bounds, showing that these rates cannot be improved upon. Our proof relies on a novel decomposition of the excess risk for general ERM-based classifiers, which may be of independent interest.

Submission Type: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=GxNDsSpNGx

Changes Since Last Submission: fixed formatting (font and margin sizes)

Assigned Action Editor: ~Nishant_A_Mehta1

Submission Number: 6138