Fourier Prior-Based Two-Stage Architecture for Image Restoration

Hemkant Nehete, Amit Monga, Partha Kaushik, Brajesh Kumar Kaushik

Abstract: This work presents a novel two stage architecture designed to enhance degraded images affected by environmental factors such as haze, blur, fog, and rain. Despite the dominance of deep Convolutional Neural Networks (CNNs) and Transformers in single image restoration tasks, existing methods neglect the intrinsic priors for physical properties of degradation. To enhance the generalization ability of image restoration models, we propose Fourier prior based on a key observation that substituting the Fourier amplitude of degraded images with that of clean images effectively mitigates degradation. Therefore, amplitude contains degradation information, while the phase retains background structures. Consequently, a two-stage model is proposed, that consists of Amplitude Refinement Unit (ARU) and the Phase Refinement Unit (PRU), that separately restore both amplitude and phase information, respectively. ARU and PRU leverage a CNN-Transformer-based architecture to extract local and global features, overcoming computational constraints posed by large image sizes in Transformers. Additionally, a multi-scale approach in ARU refines amplitude features at coarse and fine levels, improving restoration efficiency. Experimental results across multiple image restoration tasks, like image deraining, dehazing, and low-light enhancement, indicate that the proposed architecture improved the performance in terms of PSNR, SSIM, and computational efficiency compared to state-of-the-art Transformer approaches.