A Characterization Theorem for Equivariant Networks with Point-wise Activations

Marco Pacini, Xiaowen Dong, Bruno Lepri, Gabriele Santin

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Geometric Deep Learning, Equivariant Neural Networks, Characterization Theorem, Point-wise Activations

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: A characterization theorem describing admissibile combinations of representations and activation functions for equivariant layers with point-wise activations

Abstract:

Equivariant neural networks have shown improved performance, expressiveness and sample complexity on symmetrical domains.
But for some specific symmetries, representations, and choice of coordinates, the most common point-wise activations, such as ReLU, are not equivariant, hence they cannot be employed in the design of equivariant neural networks.
The theorem we present in this paper describes all possibile combinations of representations, choice of coordinates and point-wise activations to obtain an equivariant layer, generalizing and strengthening existing characterizations.
Notable cases of practical relevance are discussed as corollaries. Indeed, we prove that rotation-equivariant networks can only be invariant, as it happens for any network which is equivariant with respect to connected compact groups. Then, we discuss implications of our findings when applied to important instances of equivariant networks. First, we completely characterize permutation equivariant networks such as Invariant Graph Networks with point-wise nonlinearities and their geometric counterparts, highlighting a plethora of models whose expressive power and performance are still unknown.
Second, we show that feature spaces of disentangled steerable convolutional neural networks are trivial representations.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Primary Area: learning on graphs and other geometries & topologies

Submission Number: 6118