SO(3)-Equivariant Representation Learning in 2D Images

Published: 29 Nov 2023, Last Modified: 29 Nov 2023NeurReps 2023 PosterEveryoneRevisionsBibTeX
Submission Track: Extended Abstract
Keywords: equivariance, group convolution, deep learning, object detection, cryoEM
TL;DR: We introduce group convolutional layers that achieve 3D equivariance in 2D images via 2D projections of 3D filters, and novel means of aggregating the rotation-specific outputs.
Abstract: Imaging physical objects that are free to rotate and translate in 3D is challenging. While an object’s pose and location do not change its nature, varying them presents problems for current vision models. Equivariant models account for these nuisance transformations, but current architectures only model either 2D transformations of 2D signals or 3D trans- formations of 3D signals. Here, we propose a novel convolutional layer consisting of 2D projections of 3D filters that models 3D equivariances of 2D signals—critical for capturing the full space of spatial transformations of objects in imaging domains such as cryo-EM. We additionally present methods for aggregating our rotation-specific outputs. We demonstrate improvement on several tasks, including particle picking and pose estimation.
Submission Number: 57