Point-wise Activations and Steerable Convolutional Networks

Marco Pacini, Xiaowen Dong, Bruno Lepri, Gabriele Santin

Keywords: Deep Learning, Equivariant Neural Networks, Steerable Neural Networks

TL;DR: Characterization of steerable neural networks with point-wise activation and implications in relevant scenarios

Abstract:

Steerable Convolutional Neural Networks are a popular and efficient class of equivariant models.
For some specific groups, representations, and choice of coordinates, the most common point-wise activations, such as ReLU, are not equivariant.
Hence they cannot be employed in designing equivariant neural networks.
In this paper, we present a simple yet effective generalization of such results for equivariant networks.
First, we prove that for groups such point-wise activations can be employed in disentangled layers only when a simple group-theoretic condition is satisfied, namely when the linear representations underlying their feature spaces are trivial representations.
Second, we show analogous results for connected compact groups, where the only admitted equivariant neural networks with point-wise activations are the invariant ones.

These results demonstrate the necessity of further research for the design of suitable activation functions beyond point-wise ones.

Submission Type: Extended abstract (max 4 main pages).

Agreement: Check this if you are okay with being contacted to participate in an anonymous survey.

Submission Number: 18