Video-Based Precipitation Intensity Recognition Using Dual-Dimension and Dual-Scale Spatiotemporal Convolutional Neural Network

Chih-Wei Lin, Zhongsheng Chen, Xiuping Huang, Suhui Yang

Abstract: This paper proposes the dual-dimension and dual-scale spatiotemporal convolutional neural network, namely DDS-CNN, which consists of two modules, the global spatiotemporal module (GSM) and the local spatiotemporal module (LSM), for precipitation intensity recognition. The GSM uses 3D LSTM operations to study the influence of the relationship between sampling points on precipitation. The LSM takes 4D convolution operations and forms the convolution branches with various convolution kernels to learn the rain pattern of different precipitation. We evaluate the performance of DDS-CNN using the self-collected dataset, IMLab-RAIN-2018, and compare it with the state-of-the-art 3D models. DDS-CNN has the highest overall accuracy and achieves 98.63%. Moreover, we execute the ablation experiments to prove the effectiveness of the proposed modules.