Do ReLU Networks Have An Edge When Approximating Compactly-Supported Functions?
Abstract: We study the problem of approximating compactly-supported integrable functions while implementing their support set using feedforward neural networks. Our first main result transcribes this ``structured'' approximation problem into a universality problem. We do this by constructing a refinement of the usual topology on the space $L^1_{\operatorname{loc}}(\mathbb{R}^d,\mathbb{R}^D)$ of locally-integrable functions in which compactly-supported functions can only be approximated in $L^1$-norm by functions with matching discretized support. We establish the universality of ReLU feedforward networks with bilinear pooling layers in this refined topology. Consequentially, we find that ReLU feedforward networks with bilinear pooling can approximate compactly supported functions while implementing their discretized support. We derive a quantitative uniform version of our universal approximation theorem on the dense subclass of compactly-supported Lipschitz functions. This quantitative result expresses the depth, width, and the number of bilinear pooling layers required to construct this ReLU network via the target function's regularity, the metric capacity and diameter of its essential support, and the dimensions of the inputs and output spaces. Conversely, we show that polynomial regressors and analytic feedforward networks are not universal in this space.
Submission Length: Long submission (more than 12 pages of main content)
Changes Since Last Submission: **Main Changes**
We implemented the changes discussed with the action editor and inspired by the discussion with Reviewer 7i2i.
These are a rewriting of the paper's title, abstract, introduction, and conclusion to best reflect our main contribution: namely that ReLU networks with bilinear pooling can approximate essentially compactly supported and locally (Lebesgue) integrable functions while also implementing their support.
As discussed, we note that, there are no changes to the paper's main results only an alignment of their presentation/story with their content.
Assigned Action Editor: ~Daniel_M_Roy1
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Number: 62
Loading