Approximation, Estimation and Optimization Errors for a Deep Neural Network
Abstract: The error of supervised learning is typically split into three components: approximation, estimation and optimization errors. While all three have been extensively studied in the literature, a unified treatment is less frequent, in part because of conflicting assumptions. Current approximation results rely on carefully hand crafted weights or practically unavailable information, which are difficult to achieve by gradient descent. Optimization theory is best understood in over-parametrized regimes with more weights than samples, while classical estimation errors require the opposite regime with more samples than weights.
This paper contains two results which bound all three error components simultaneously for deep fully connected networks. The first uses a regular least squares loss and shows convergence in the under-parametrized regime. The second uses a kernel based loss function and shows convergence in both under and over-parametrized regimes.
Submission Length: Long submission (more than 12 pages of main content)
Previous TMLR Submission Url: https://openreview.net/forum?id=nnTKcGNrbV
Changes Since Last Submission: We made the following changes suggested by the editor of the previous submission:
0. Section 1.2 "New Contributions" has been rewritten based on the editors suggestions.
1. We removed all indented bullets form the literature review Section 1.1 and overview Section 1.3.
2. We added a conclusion.
3. Section 3 " Numerical Experiments" has been rewritten. It now includes comparisons with the proven error bounds as well as theoretically optimal bounds.
4. We changed the notation in Lemma B.4, as requested.
Assigned Action Editor: ~Martha_White1
Submission Number: 3868
Loading