A Resource-Constrained Spatio-Temporal Super Resolution Model

Da Hyeon Jung, Min-Wu Jeong, Xuan Truong Nguyen, Chae-Eun Rhee

Abstract: This paper proposes a resource-constrained spatio-temporal super resolution (SR) model, which effectively enhances both the spatial resolution and frame rate of input videos. Replacing the entire deep learning model for spatio-temporal SR on devices that already have spatial SR capability is a challenging task. This is especially true for compact devices like image sensors that are composed of hardware modules. There is a need to enable spatio-temporal SR with minimal hardware overhead on devices that already have the SR module. The proposed model demonstrates an example of hardware implementation by combining independent hardware-friendly spatial SR and frame interpolation (FI). This configuration allows for seamless support of spatial SR, temporal SR, and spatio-temporal SR functionalities through data flow reconfiguration. Moreover, we propose schemes that leverage the flow estimation module to further reduce the computational burden of spatial SR. The experimental results show that the proposed model achieves competitive quality with state-of-the-art (SOTA) methods, while utilizing very limited computational resources.