Nonlinear approximation via compositions

Zuowei Shen, Haizhao Yang, Shijun Zhang

Abstract: Given a function dictionary D<math><mi mathvariant="script" is="true">D</mi></math> and an approximation budget N∈N<math><mrow is="true"><mi is="true">N</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">∈</mo><mi mathvariant="double-struck" is="true">N</mi></mrow></math>, nonlinear approximation seeks the linear combination of the best N<math><mi is="true">N</mi></math> terms {Tn}1≤n≤N⊆D<math><mrow is="true"><msub is="true"><mrow is="true"><mrow is="true"><mo is="true">{</mo><msub is="true"><mrow is="true"><mi is="true">T</mi></mrow><mrow is="true"><mi is="true">n</mi></mrow></msub><mo is="true">}</mo></mrow></mrow><mrow is="true"><mn is="true">1</mn><mo is="true">≤</mo><mi is="true">n</mi><mo is="true">≤</mo><mi is="true">N</mi></mrow></msub><mo linebreak="goodbreak" linebreakstyle="after" is="true">⊆</mo><mi mathvariant="script" is="true">D</mi></mrow></math> to approximate a given function f<math><mi is="true">f</mi></math> with the minimum approximation error εL,f≔min{gn}⊆R,{Tn}⊆D‖f(x)−∑n=1NgnTn(x)‖.<math><mrow is="true"><msub is="true"><mrow is="true"><mi is="true">ε</mi></mrow><mrow is="true"><mi is="true">L</mi><mo is="true">,</mo><mi is="true">f</mi></mrow></msub><mo linebreak="goodbreak" is="true">≔</mo><munder is="true"><mrow is="true"><mo class="qopname" is="true">min</mo></mrow><mrow is="true"><mrow is="true"><mo is="true">{</mo><msub is="true"><mrow is="true"><mi is="true">g</mi></mrow><mrow is="true"><mi is="true">n</mi></mrow></msub><mo is="true">}</mo></mrow><mo linebreak="badbreak" is="true">⊆</mo><mi mathvariant="double-struck" is="true">R</mi><mo is="true">,</mo><mrow is="true"><mo is="true">{</mo><msub is="true"><mrow is="true"><mi is="true">T</mi></mrow><mrow is="true"><mi is="true">n</mi></mrow></msub><mo is="true">}</mo></mrow><mo linebreak="badbreak" is="true">⊆</mo><mi mathvariant="script" is="true">D</mi></mrow></munder><mo is="true">‖</mo><mi is="true">f</mi><mrow is="true"><mo is="true">(</mo><mi mathvariant="bold-italic" is="true">x</mi><mo is="true">)</mo></mrow><mo linebreak="goodbreak" is="true">−</mo><munderover is="true"><mrow is="true"><mo linebreak="badbreak" is="true">∑</mo></mrow><mrow is="true"><mi is="true">n</mi><mo linebreak="badbreak" is="true">=</mo><mn is="true">1</mn></mrow><mrow is="true"><mi is="true">N</mi></mrow></munderover><msub is="true"><mrow is="true"><mi is="true">g</mi></mrow><mrow is="true"><mi is="true">n</mi></mrow></msub><msub is="true"><mrow is="true"><mi is="true">T</mi></mrow><mrow is="true"><mi is="true">n</mi></mrow></msub><mrow is="true"><mo is="true">(</mo><mi mathvariant="bold-italic" is="true">x</mi><mo is="true">)</mo></mrow><mo is="true">‖</mo><mo is="true">.</mo></mrow></math>Motivated by recent success of deep learning, we propose dictionaries with functions in a form of compositions, i.e., T(x)=T(L)∘T(L−1)∘⋯∘T(1)(x)<math><mrow is="true"><mi is="true">T</mi><mrow is="true"><mo is="true">(</mo><mi mathvariant="bold-italic" is="true">x</mi><mo is="true">)</mo></mrow><mo linebreak="goodbreak" is="true">=</mo><msup is="true"><mrow is="true"><mi is="true">T</mi></mrow><mrow is="true"><mrow is="true"><mo is="true">(</mo><mi is="true">L</mi><mo is="true">)</mo></mrow></mrow></msup><mo is="true">∘</mo><msup is="true"><mrow is="true"><mi is="true">T</mi></mrow><mrow is="true"><mrow is="true"><mo is="true">(</mo><mi is="true">L</mi><mo linebreak="badbreak" is="true">−</mo><mn is="true">1</mn><mo is="true">)</mo></mrow></mrow></msup><mo is="true">∘</mo><mo linebreak="goodbreak" is="true">⋯</mo><mo is="true">∘</mo><msup is="true"><mrow is="true"><mi is="true">T</mi></mrow><mrow is="true"><mrow is="true"><mo is="true">(</mo><mn is="true">1</mn><mo is="true">)</mo></mrow></mrow></msup><mrow is="true"><mo is="true">(</mo><mi mathvariant="bold-italic" is="true">x</mi><mo is="true">)</mo></mrow></mrow></math>for all T∈D<math><mrow is="true"><mi is="true">T</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">∈</mo><mi mathvariant="script" is="true">D</mi></mrow></math>, and implement T<math><mi is="true">T</mi></math> using ReLU feed-forward neural networks (FNNs) with L<math><mi is="true">L</mi></math> hidden layers. We further quantify the improvement of the best N<math><mi is="true">N</mi></math>-term approximation rate in terms of N<math><mi is="true">N</mi></math> when L<math><mi is="true">L</mi></math> is increased from 1 to 2 or 3 to show the power of compositions. In the case when L>3<math><mrow is="true"><mi is="true">L</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">&gt;</mo><mn is="true">3</mn></mrow></math>, our analysis shows that increasing L<math><mi is="true">L</mi></math> cannot improve the approximation rate in terms of N<math><mi is="true">N</mi></math>.In particular, for any function f<math><mi is="true">f</mi></math> on [0,1]<math><mrow is="true"><mo is="true">[</mo><mn is="true">0</mn><mo is="true">,</mo><mn is="true">1</mn><mo is="true">]</mo></mrow></math>, regardless of its smoothness and even the continuity, if f<math><mi is="true">f</mi></math> can be approximated using a dictionary when L=1<math><mrow is="true"><mi is="true">L</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">=</mo><mn is="true">1</mn></mrow></math> with the best N<math><mi is="true">N</mi></math>-term approximation rate εL,f=O(N−η)<math><mrow is="true"><msub is="true"><mrow is="true"><mi is="true">ε</mi></mrow><mrow is="true"><mi is="true">L</mi><mo is="true">,</mo><mi is="true">f</mi></mrow></msub><mo linebreak="goodbreak" linebreakstyle="after" is="true">=</mo><mi mathvariant="script" is="true">O</mi><mrow is="true"><mo is="true">(</mo><msup is="true"><mrow is="true"><mi is="true">N</mi></mrow><mrow is="true"><mo is="true">−</mo><mi is="true">η</mi></mrow></msup><mo is="true">)</mo></mrow></mrow></math>, we show that dictionaries with L=2<math><mrow is="true"><mi is="true">L</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">=</mo><mn is="true">2</mn></mrow></math> can improve the best N<math><mi is="true">N</mi></math>-term approximation rate to εL,f=O(N−2η)<math><mrow is="true"><msub is="true"><mrow is="true"><mi is="true">ε</mi></mrow><mrow is="true"><mi is="true">L</mi><mo is="true">,</mo><mi is="true">f</mi></mrow></msub><mo linebreak="goodbreak" linebreakstyle="after" is="true">=</mo><mi mathvariant="script" is="true">O</mi><mrow is="true"><mo is="true">(</mo><msup is="true"><mrow is="true"><mi is="true">N</mi></mrow><mrow is="true"><mo is="true">−</mo><mn is="true">2</mn><mi is="true">η</mi></mrow></msup><mo is="true">)</mo></mrow></mrow></math>. We also show that for Hölder continuous functions of order α<math><mi is="true">α</mi></math> on [0,1]d<math><msup is="true"><mrow is="true"><mrow is="true"><mo is="true">[</mo><mn is="true">0</mn><mo is="true">,</mo><mn is="true">1</mn><mo is="true">]</mo></mrow></mrow><mrow is="true"><mi is="true">d</mi></mrow></msup></math>, the application of a dictionary with L=3<math><mrow is="true"><mi is="true">L</mi><mo linebreak="goodbreak" linebreakstyle="after" is="true">=</mo><mn is="true">3</mn></mrow></math> in nonlinear approximation can achieve an essentially tight best N<math><mi is="true">N</mi></math>-term approximation rate εL,f=O(N−2α∕d)<math><mrow is="true"><msub is="true"><mrow is="true"><mi is="true">ε</mi></mrow><mrow is="true"><mi is="true">L</mi><mo is="true">,</mo><mi is="true">f</mi></mrow></msub><mo linebreak="goodbreak" linebreakstyle="after" is="true">=</mo><mi mathvariant="script" is="true">O</mi><mrow is="true"><mo is="true">(</mo><msup is="true"><mrow is="true"><mi is="true">N</mi></mrow><mrow is="true"><mo is="true">−</mo><mn is="true">2</mn><mi is="true">α</mi><mo is="true">∕</mo><mi is="true">d</mi></mrow></msup><mo is="true">)</mo></mrow></mrow></math>. Finally, we show that dictionaries consisting of wide FNNs with a few hidden layers are more attractive in terms of computational efficiency than dictionaries with narrow and very deep FNNs for approximating Hölder continuous functions if the number of computer cores is larger than N<math><mi is="true">N</mi></math> in parallel computing.