What Survives of Path Norms? Path-Lifting as an Intermediate Representation for ReLU Networks

Antoine Gonon, Rémi Gribonval

Keywords: ReLU networks, symmetries, path-lifting, path-activations, path-norms

Abstract:

More than a decade ago, Neyshabur et al. introduced path norms to define complexity measures over classes of functions implemented by ReLU networks, tightening existing bounds by factoring out intrinsic rescaling-invariances of the weight-space parameterization of such networks. While conceptually exciting, this however did not fully match expectations, as path norms bounds often remain several orders of magnitude too large to provide stand-alone quantitative bounds, *e.g.*, on generalization error or Lipschitz constants.

Path-norms are however only the most visible facet of a toolset built on top of *path-lifting* and *path-activations*, two complementary rescaling-invariant representations of ReLU networks.
This short perspective paper brings these layers back to the foreground: the lasting contribution of the introduction of path norms is not a single privileged---but overly pessimistic---scalar complexity measure, but a representational toolset for formulating weight-space questions after rescaling has been factored out.
As we highlight, this toolset
keeps reappearing in invariant embeddings for identifiability, symmetry-aware optimization, conservation laws for gradient flow, pruning, and recent PAC-Bayes analyses, to name a few.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 13