Scalable Super-Resolution Neural Operator

Lei Han, Xuesong Zhang

Abstract:

Recent advances in continuous super-resolution (SR) has made a substantial progress towards universal SR models, which are characterized by using a single deep neural network (DNN) to fulfill arbitrary scale SR tasks. When deployed on resource stringent platforms, however, a trained DNN model usually requires experience-demanding and laborious manual efforts to compress the models following a predetermined compute budget. This paper proposes an inference-time adaptive network width optimization method for arbitrary scale SR modules, dubbed as Scalable Super-Resolution Neural Operator (SSRNO), which is capable of efficient performance-preserving deployment on various mobile or edge devices with only a user input parameter indicating the desired compression rate. SSRNO realizes the continuous parameterization of SRNO(CVPR2023) by virtue of two novel contributions. First, we propose the Integral Neural Network (INN) formulation for the Galerkin type attention, which is an indispensable component for spatial discretization invariant SR neural networks. Second, we further propose an adaptive layer-wise compression rate estimation mechanism, which allows for the flexible adaptation to variant capacity through the neural network layers. Extensive experiments validate the outperforming overall performances over existing continuous SR models in terms of reconstruction accuracy, model scalability as well as throughput. For instance, compared with the baseline SRNO, a typical configuration of SSRNO can achieve a model size compression up to 62% and an over 2$\times$ speedup in situations where resources are limited, while it can also expand itself to keep the PSNR degradation within 0.1 dBs when the limitations are alleviated. The code will be made public soon.

Primary Subject Area: [Experience] Multimedia Applications

Relevance To Conference: Super-resolution, as a classic problem in computer vision, has widespread applications in various fields. However, most of current SR models are fixed in model size and have significant limitations in adapting to different tasks and edge devices. In our paper, we propose a scalable continuous super-resolution model that can adjust its resource requirements to accommodate constraints of different tasks and devices, enabling it to provide SR services effectively. Our work has contributed to the lightweighting of models processing multimedia data and their application on edge devices. We have introduced a novel objective for model lightweighting and mobilization, which is to adapt to the requirements of different platforms and provide corresponding experiences based on computational resources. In such cases, users can specify performance requirements as needed or specify the computational power requirements of the model. This helps to conserve resources, which are already limited on edge devices and embedded platforms.

Submission Number: 3629