Keywords: Point cloud, spatio-temporal modeling, video analysis, action recognition, semantic segmentation, convolutional neural network
Abstract: Point cloud sequences are irregular and unordered in the spatial dimension while exhibiting regularities and order in the temporal dimension. Therefore, existing grid based convolutions for conventional video processing cannot be directly applied to spatio-temporal modeling of raw point cloud sequences. In this paper, we propose a point spatio-temporal (PST) convolution to achieve informative representations of point cloud sequences. The proposed PST convolution first disentangles space and time in point cloud sequences. Then, a spatial convolution is employed to capture the local structure of points in the 3D space, and a temporal convolution is used to model the dynamics of the spatial regions along the time dimension. Furthermore, we incorporate the proposed PST convolution into a deep network, namely PSTNet, to extract features of point cloud sequences in a hierarchical manner. Extensive experiments on widely-used 3D action recognition and 4D semantic segmentation datasets demonstrate the effectiveness of PSTNet to model point cloud sequences.
One-sentence Summary: This paper proposes a point spatio-temporal (PST) convolution, which decomposes space and time, to learn representations of raw point cloud sequences in a spatio-temporally hierarchical manner.
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics
Code: [![github](/images/github_icon.svg) hehefan/Point-Spatio-Temporal-Convolution](https://github.com/hehefan/Point-Spatio-Temporal-Convolution)
Data: [NTU RGB+D](https://paperswithcode.com/dataset/ntu-rgb-d), [SYNTHIA](https://paperswithcode.com/dataset/synthia)