Reference-based Burst Super-resolution

Seonggwan Ko, Yeong Jun Koh, Donghyeon Cho

Abstract: Burst super-resolution (BurstSR) utilizes signal information from multiple adjacent frames successively taken to restore rich textures. However, due to hand tremors and other image degradation factors, even recent BurstSR methods struggle to reconstruct finely textured images. On the other hand, reference-based super-resolution (RefSR) leverages the high-fidelity reference (Ref) image to recover detailed contents. Nevertheless, if there is no correspondence between the Ref and the low-resolution (LR) image, the degraded output is derived. To overcome the limitations of existing BurstSR and RefSR methods, we newly introduce a reference-based burst super-resolution (RefBSR) that utilizes burst frames and a high-resolution (HR) external Ref image. The RefBSR can restore the HR image by properly fusing the benefits of burst frames and a Ref image. To this end, we propose the first RefBSR framework that consists of Ref-burst feature matching and burst feature-aware Ref texture transfer (BRTT) modules. In addition, our method adaptively integrates features with better quality between Ref and burst features using Ref-burst adaptive feature fusion (RBAF). To train and evaluate our method, we provide a new dataset of Ref-burst pairs collected by commercial smartphones. The proposed method achieves state-of-the-art performance compared to both existing RefSR and BurstSR methods, and we demonstrate its effectiveness through comprehensive experiments. The source codes and the newly constructed dataset will be made publicly available for further research.

Primary Subject Area: [Content] Vision and Language

Secondary Subject Area: [Experience] Multimedia Applications

Relevance To Conference: This paper is a study on fusing images from dual cameras with different characteristics, and is highly related to multimodal processing and multimedia applications for mobile devices and multi-camera systems.

Submission Number: 450