Convolutional Neural Network-Based Motion Compensation Refinement for Video Coding

Shuai Huo, Dong Liu, Feng Wu, Houqiang Li

Abstract: Inspired by the great success of convolutional neural network (CNN) in computer vision, we propose a CNN-based method to refine the motion compensation in video coding. First, we study a simple CNN-based motion compensation refinement (CNNMCR) scheme, where we train a CNN to refine the motion compensated prediction directly. Second, we consider to exploit the contextual information for the refinement, and propose a more powerful CNNMCR scheme, where the CNN utilizes not only the motion compensated prediction, but also the neighboring reconstructed region to refine the prediction. We integrate the simple CNNMCR and the CNNMCR schemes into the High Efficiency Video Coding (HEVC) framework. Experimental results show that both schemes achieve better compression performance than the HEVC anchor, leading to on average 1.8% and 2.3% BD-rate reduction, respectively, under low-delay P configuration. Furthermore, the combination of our proposed CNNMCR and the overlapped block motion compensation (OBMC) technique provides as high as 5.2% BD-rate reduction.