An Energy-Efficient Differential Frame Convolutional Accelerator with on-Chip Fusion Storage Architecture and Pixel-Level Pipeline Data Flow

Zhenhui Dai, Jiawei Wang, Yi Zhong, Kunyu Feng, Cheng Zhao, Yuanyuan Jiang, Peiyu Chen, Yuan Wang, Dunshan Yu, Xiaoxin Cui

Abstract: Convolutional neural networks require a huge amount of computation in video applications. For some specific tasks, such as surveillance, differential frame convolution reuses inter-frame data and significantly reduces multiplication and accumulation. However, there are still some challenges in improving energy efficiency of differential frame convolution on chips. Firstly, differential frame convolution brings additional on-chip storage for reusing inter-frame data. Secondly, in post-processing of differential frame convolution, there are more memory accessing and arithmetic logic operations. Therefore, sparse working mode is of vital importance for the post-processing. In response to these challenges, this work proposes an on-chip fusion storage architecture for energy-efficient differential frame convolution and a pixel-level pipeline data flow that supports the sparsity of features. The simulation of our accelerator implemented in 28nm CMOS can achieve energy efficiency by 3.09× compared with other state-of-the-art works