Schauder Bases for $C[0, 1]$ Using ReLU, Softplus and Two Sigmoidal Functions

TMLR Paper5132 Authors

Abstract: We construct four Schauder bases for the space $C[0,1]$, one using ReLU functions, another using Softplus functions, and two more using sigmoidal versions of the ReLU and Softplus functions. This establishes the existence of a basis using these functions for the first time, and improves on the universal approximation property associated with them. We also show an $O(\frac{1}{n})$ approximation bound based on our ReLU basis, and a negative result on constructing multivariate functions using finite combinations of ReLU functions.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

Camera Ready Changes:
- Deanonymization and acknowledgement section
- Add function plots in the appendix for the ReLU function, the first differences, the second differences, the counter example, a perturbed basis element and the pyramidal function.
- Minor rewording of some introductory text clarifying the use of function composition similar to a two-layer network in the Kolmogorov approach

Changes:
- New result (Theorem 7) on $O(\frac{1}{n})$ supnorm approximation using interpolation.
- New negative result (Theorem 8) for multidimensional input using finite linear combinations of ReLU functions, indicating the possible necessity of multilayer networks. We are hoping that we can generalize this further (soon!) to countable linear combinations, which will negate the possibility of a Schauder basis using ReLU functions for multidimensional input.
- Add some text before the proof of Lemma 1 separating out the routine and subtle portions of the proof, along with a counter example for general (or trivial) linear combinations of piecewise linear functions.
- Update introduction: connect our basis results to approximation bounds, both our own and similar ones in literature.
- Add a few plots in the appendix, but this is work in progress. We wanted to clarify a couple of points before we finalize these plots.
- Fix typo error $r(\frac{1}{2}) = \frac{1}{2}$ as described earlier, with renormalization of $s_{n, k}$ to $\frac{1}{2}$ as well.

Assigned Action Editor: ~Surbhi_Goel1

Submission Number: 5132