ISP2DLA: Automated Deep Learning Accelerator Design for On-Sensor Image Signal Processing

Dong-eon Won, Yeeun Kim, Janghwan Lee, Minjae Lee, Jonghyun Bae, Jongjoo Park, Jeongyong Song, Jungwook Choi

Abstract: Deep neural network-based image signal processing (ISP-DNN) improves image quality with techniques such as demosaicing, but these models pose substantial computational and memory challenges when implemented on CMOS image sensors, particularly due to the high-resolution inputs that increase memory requirements for activations. Layer fusion reduces memory usage by combining consecutive processing steps, yet it increases computational demands, a critical issue in resource-limited on-sensor environments. To address these challenges, we introduce ISP2DLA, an automated deep learning accelerator design framework that balances computational and memory demands for on-sensor ISP. This framework optimizes hardware designs by adjusting line buffer sizes and the number of MAC units, reducing gate counts by 14-79% across two ISP-DNN models, thus enabling efficient on-sensor ISP model inference within constrained resources.