Simultaneous neural network approximation for smooth functions

Sean Hon, Haizhao Yang

Abstract: We establish in this work approximation results of deep neural networks for smooth functions measured in Sobolev norms, motivated by recent development of numerical solvers for partial differential equations using deep neural networks. Our approximation results are nonasymptotic in the sense that the error bounds are explicitly characterized in terms of both the width and depth of the networks simultaneously with all involved constants explicitly determined. Namely, for f∈Cs([0,1]d)<math><mrow is="true"><mi is="true">f</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">∈</mo><msup is="true"><mrow is="true"><mi is="true">C</mi></mrow><mrow is="true"><mi is="true">s</mi></mrow></msup><mrow is="true"><mo is="true">(</mo><msup is="true"><mrow is="true"><mrow is="true"><mo is="true">[</mo><mn is="true">0</mn><mo is="true">,</mo><mn is="true">1</mn><mo is="true">]</mo></mrow></mrow><mrow is="true"><mi is="true">d</mi></mrow></msup><mo is="true">)</mo></mrow></mrow></math>, we show that deep ReLU networks of width O(NlogN)<math><mrow is="true"><mi mathvariant="script" is="true">O</mi><mrow is="true"><mo is="true">(</mo><mi is="true">N</mi><mo class="qopname" is="true">log</mo><mi is="true">N</mi><mo is="true">)</mo></mrow></mrow></math> and of depth O(LlogL)<math><mrow is="true"><mi mathvariant="script" is="true">O</mi><mrow is="true"><mo is="true">(</mo><mi is="true">L</mi><mo class="qopname" is="true">log</mo><mi is="true">L</mi><mo is="true">)</mo></mrow></mrow></math> can achieve a nonasymptotic approximation rate of O(N−2(s−1)/dL−2(s−1)/d)<math><mrow is="true"><mi mathvariant="script" is="true">O</mi><mrow is="true"><mo is="true">(</mo><msup is="true"><mrow is="true"><mi is="true">N</mi></mrow><mrow is="true"><mo is="true">−</mo><mn is="true">2</mn><mrow is="true"><mo is="true">(</mo><mi is="true">s</mi><mo is="true">−</mo><mn is="true">1</mn><mo is="true">)</mo></mrow><mo is="true">/</mo><mi is="true">d</mi></mrow></msup><msup is="true"><mrow is="true"><mi is="true">L</mi></mrow><mrow is="true"><mo is="true">−</mo><mn is="true">2</mn><mrow is="true"><mo is="true">(</mo><mi is="true">s</mi><mo is="true">−</mo><mn is="true">1</mn><mo is="true">)</mo></mrow><mo is="true">/</mo><mi is="true">d</mi></mrow></msup><mo is="true">)</mo></mrow></mrow></math> with respect to the W1,p([0,1]d)<math><mrow is="true"><msup is="true"><mrow is="true"><mi mathvariant="script" is="true">W</mi></mrow><mrow is="true"><mn is="true">1</mn><mo is="true">,</mo><mi is="true">p</mi></mrow></msup><mrow is="true"><mo is="true">(</mo><msup is="true"><mrow is="true"><mrow is="true"><mo is="true">[</mo><mn is="true">0</mn><mo is="true">,</mo><mn is="true">1</mn><mo is="true">]</mo></mrow></mrow><mrow is="true"><mi is="true">d</mi></mrow></msup><mo is="true">)</mo></mrow></mrow></math> norm for p∈[1,∞)<math><mrow is="true"><mi is="true">p</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">∈</mo><mrow is="true"><mo is="true">[</mo><mn is="true">1</mn><mo is="true">,</mo><mi is="true">∞</mi><mo is="true">)</mo></mrow></mrow></math>. If either the ReLU function or its square is applied as activation functions to construct deep neural networks of width O(NlogN)<math><mrow is="true"><mi mathvariant="script" is="true">O</mi><mrow is="true"><mo is="true">(</mo><mi is="true">N</mi><mo class="qopname" is="true">log</mo><mi is="true">N</mi><mo is="true">)</mo></mrow></mrow></math> and of depth O(LlogL)<math><mrow is="true"><mi mathvariant="script" is="true">O</mi><mrow is="true"><mo is="true">(</mo><mi is="true">L</mi><mo class="qopname" is="true">log</mo><mi is="true">L</mi><mo is="true">)</mo></mrow></mrow></math> to approximate f∈Cs([0,1]d)<math><mrow is="true"><mi is="true">f</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">∈</mo><msup is="true"><mrow is="true"><mi is="true">C</mi></mrow><mrow is="true"><mi is="true">s</mi></mrow></msup><mrow is="true"><mo is="true">(</mo><msup is="true"><mrow is="true"><mrow is="true"><mo is="true">[</mo><mn is="true">0</mn><mo is="true">,</mo><mn is="true">1</mn><mo is="true">]</mo></mrow></mrow><mrow is="true"><mi is="true">d</mi></mrow></msup><mo is="true">)</mo></mrow></mrow></math>, the approximation rate is O(N−2(s−n)/dL−2(s−n)/d)<math><mrow is="true"><mi mathvariant="script" is="true">O</mi><mrow is="true"><mo is="true">(</mo><msup is="true"><mrow is="true"><mi is="true">N</mi></mrow><mrow is="true"><mo is="true">−</mo><mn is="true">2</mn><mrow is="true"><mo is="true">(</mo><mi is="true">s</mi><mo is="true">−</mo><mi is="true">n</mi><mo is="true">)</mo></mrow><mo is="true">/</mo><mi is="true">d</mi></mrow></msup><msup is="true"><mrow is="true"><mi is="true">L</mi></mrow><mrow is="true"><mo is="true">−</mo><mn is="true">2</mn><mrow is="true"><mo is="true">(</mo><mi is="true">s</mi><mo is="true">−</mo><mi is="true">n</mi><mo is="true">)</mo></mrow><mo is="true">/</mo><mi is="true">d</mi></mrow></msup><mo is="true">)</mo></mrow></mrow></math> with respect to the Wn,p([0,1]d)<math><mrow is="true"><msup is="true"><mrow is="true"><mi mathvariant="script" is="true">W</mi></mrow><mrow is="true"><mi is="true">n</mi><mo is="true">,</mo><mi is="true">p</mi></mrow></msup><mrow is="true"><mo is="true">(</mo><msup is="true"><mrow is="true"><mrow is="true"><mo is="true">[</mo><mn is="true">0</mn><mo is="true">,</mo><mn is="true">1</mn><mo is="true">]</mo></mrow></mrow><mrow is="true"><mi is="true">d</mi></mrow></msup><mo is="true">)</mo></mrow></mrow></math> norm for p∈[1,∞)<math><mrow is="true"><mi is="true">p</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">∈</mo><mrow is="true"><mo is="true">[</mo><mn is="true">1</mn><mo is="true">,</mo><mi is="true">∞</mi><mo is="true">)</mo></mrow></mrow></math>.