3D Point Cloud Sequences as 2D Videos

YIMING ZENG, Junhui Hou, Qijian Zhang, Wenping Wang

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: 3D point cloud

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: A novel representation modality for efficient and effective deep learning-based processing and analysis of point cloud sequences, which will opens up new possibilities.

Abstract: The irregular and unstructured nature of 3D point cloud sequences in both spatial and temporal domains poses great difficulties in extracting their discriminative features effectively and efficiently. To tackle these challenges, in contrast to existing methods devoted to developing special architectures for modeling sequences, we advocate a new paradigm by proposing a novel representation modality, called point geometry video (PGV), that encodes the coordinates of the 3D points of a sequence as the pixel values of a 2D color video, with the original spatial neighborhood relationship and temporal consistency preserved. PGV significantly facilitates the processing of sequential 3D point clouds by enabling the adaption of powerful learning techniques for 2D image and video processing. Technically, by leveraging the local aggregation and kernel-sharing properties of the convolution operation, we build a self-supervised auto-encoder composed of convolutional layers, that consumes pre-defined regular grids to produce the PGV representation of a sequence of point clouds. We demonstrate the superiority and generality of the PGV on downstream tasks, including sequence correspondence, spatial upsampling, and forecasting. The PGV as a novel representation modality opens up new possibilities for deep learning-based processing and analysis of point cloud sequences. The code and data will be made publicly available.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1550