Rethinking Imbalance in Image Super-Resolution for Efficient Inference

Wei Yu, Bowen Yang, Qinglin Liu, Jianing Li, Shengping Zhang, Xiangyang Ji

Keywords: Efficient Image Super-Resolution; Dynamic Network; Imbalanced Data Learning

TL;DR: This paper explores the imbalance in the image SR task and proposes a plug-and-play Weight-Balancing framework (WBSR) based on a HES strategy and a BDLoss to achieve accurate and efficient inference.

Abstract: Existing super-resolution (SR) methods optimize all model weights equally using $\mathcal{L}_1$ or $\mathcal{L}_2$ losses by uniformly sampling image patches without considering dataset imbalances or parameter redundancy, which limits their performance. To address this, we formulate the image SR task as an imbalanced distribution transfer learning problem from a statistical probability perspective, proposing a plug-and-play Weight-Balancing framework (WBSR) to achieve balanced model learning without changing the original model structure and training data. Specifically, we develop a Hierarchical Equalization Sampling (HES) strategy to address data distribution imbalances, enabling better feature representation from texture-rich samples. To tackle model optimization imbalances, we propose a Balanced Diversity Loss (BDLoss) function, focusing on learning texture regions while disregarding redundant computations in smooth regions. After joint training of HES and BDLoss to rectify these imbalances, we present a gradient projection dynamic inference strategy to facilitate accurate and efficient inference. Extensive experiments across various models, datasets, and scale factors demonstrate that our method achieves comparable or superior performance to existing approaches with about 34\% reduction in computational cost.

Primary Area: Deep learning architectures

Submission Number: 2756