Deep Learning-Assisted Video Compression Framework

Hengyu Man, Chang Yu, Feng Xing, Yang Cheng, Bo Zheng, Xiaopeng Fan

Abstract: Deep learning has shown its great potential in video coding. Lots of research has been done regarding deep tools used within traditional schemes or end-to-end deep codec. In this paper, we propose multiple novel deep tools focusing on intra prediction, motion compensation refinement, virtual reference frame, and post-processing and form a deep learning-assisted video compression framework. For intra prediction, a novel data clustering-driven neural network (DCDNN), which could learn deep features of the clustered data, is proposed to assist intra angular modes. For inter prediction, a neural network-based motion compensation refinement (NNMCR) algorithm is applied to enhance the motion compensation. Meanwhile, a V-RF network is employed to generate a more reliable reference frame that can characterize complex motions in natural video signals. For post-processing, a quality adaptive neural network-based in-loop filter (QANNLF) is designed to utilize Quality Parameter (QP) information in the encoder to adjust filter strength for reconstructed videos with various qualities. Our method can obtain on average 14.83% BD-rate saving for the luma component under the all intra configuration and 12.16% under the random access configuration compared with HEVC reference software 16.9.