FIACCEL: Memory Efficient Frame Interpolation Accelerator for Full-HD Video

Min-Wu Jeong, Chae-Eun Rhee

Abstract: Frame interpolation (FI) is a challenging task that involves generating intermediate frames between two consecutive frames of a video to achieve smooth motion. Although several approaches, including deep learning-based and hybrid methods, have been proposed, most target GPU systems with high computational costs, making it difficult for real-time on-device systems. This brief proposes a memory-efficient and low-complexity accelerator for FI by analyzing the most memory-inefficient part of the encoder-decoder structure and applying schemes such as feature map reuse, selective transfer to DRAM, row-wise layer fusion, kernel decomposition, and parallelized horizontally dilated convolutions. The proposed hardware is verified on an FPGA environment and can synthesize $1920\times 1080$ video from 90 fps to 180 fps in real-time with an average PSNR quality of 31.98 dB on the Vimeo90K dataset. Our design is available at https://github.com/Miinuuu/FIACCEL .