ProRes: Exploring Degradation-aware Visual Prompt for Universal Image Restoration

Jiaqi Ma, Tianheng Cheng, Guoli Wang, Qian Zhang, Xinggang Wang, Lefei Zhang

Abstract:

Image restoration aims to reconstruct degraded images, e.g., denoising or deblurring. Existing works focus on designing task-specific methods and there are
inadequate attempts at universal methods. However, simply unifying multiple
tasks into one universal architecture suffers from uncontrollable and undesired
predictions. To address those issues, we explore prompt learning in universal
architectures for image restoration tasks. In this paper, we present Degradationaware Visual Prompts, which encode various types of image degradation, e.g.,
noise and blur, into unified visual prompts. These degradation-aware prompts
provide control over image processing and allow weighted combinations for customized image restoration as shown in Fig. 1. We then leverage degradationaware visual Prompts to establish a controllable and universal model for image Restoration, called ProRes, which is applicable to an extensive range of image
restoration tasks. ProRes leverages the vanilla Vision Transformer (ViT) without any task-specific designs. Furthermore, the pre-trained ProRes can easily
adapt to new tasks through efficient prompt tuning with only a few images. Without bells and whistles, ProRes achieves competitive performance compared to
task-specific methods and experiments can demonstrate its ability for controllable
restoration and adaptation for new tasks. The code and models will be released in
https://github.com/leonmakise/ProRes.