Deep Convolution for Irregularly Sampled Temporal Point Clouds

Erich Merrill III, Stefan Lee, Li Fuxin, Thomas G Dietterich, Alan Fern

Keywords: point cloud, convolution, irregular sampling, starcraft, nowcasting

Abstract:

We consider the problem of modeling the dynamics of continuous spatial-temporal processes represented by irregular samples through both space and time. Such processes occur in sensor networks, citizen science, multi-robot systems, and many others.
We propose a new deep model that is able to directly learn and predict over this irregularly sampled data, without voxelization, by leveraging a recent convolutional architecture for static point clouds. The model also easily incorporates the notion of multiple entities in the process. In particular, the model can flexibly answer prediction queries about arbitrary space-time points for different entities regardless of the distribution of the training or test-time data. We present experiments on real-world weather station data and battles between large armies in StarCraft II. The results demonstrate the model's flexibility in answering a variety of query types and demonstrate improved performance and efficiency compared to state-of-the-art baselines.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We use a set function which is equivalent to convolution to reason about irregularly sampled spatio-temporal point clouds and make predictions for arbitrary domain-specific queries.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=jLLoK91ry