DDA: A Dynamic Difficulty-aware Data Augmenter for Image Super-resolution

Xinyi Zhang, Tao Dai, Bin Chen, Shu-Tao Xia

Abstract: Deep neural networks (DNNs) have been recently widely used in image super-resolution (SR) and have achieved remarkable performance. However, most existing methods focus on elaborate network design, while rarely considering the training strategy, which affects the model performance and training efficiency. In practice, most SR methods still train the networks with the commonly-used data augmentation (e.g., random crop and sampling), which is shown to converge slowly for deep SR networks. To address this issue, in this paper, we propose a dynamic difficulty-aware data augmenter, named DDA, by considering the restoration difficulty and distribution of input patches. Our DDA mainly consists of difficulty-aware divider, dynamic sampler, and adaptive re-weighter. Specifically, our DDA first uses the difficulty-aware divider to divide the input image into small over-lapping patches, followed by classification into <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$N$</tex> different classes based on the restoration difficulty. Next, dynamic sampler samples the training patches from each class with probability based on training loss. Furthermore, to remedy the imbalance of training patches between different classes, adaptive re-weighter updates the weight of each training patch according to the accumulated training loss. Extensive experiments demonstrate the effectiveness of our DDA on different SR methods by improving training efficiency and model performance across a wide range of scenarios.