Language-driven All-in-one Adverse Weather Removal

Hao Yang, Liyuan Pan, Yan Yang, Wei Liang

Abstract: All-in-one (AiO) frameworks restore various adverse weather degradations with a single set of networks jointly. To handle various weather conditions, an AiO framework is expected to adaptively learn weather-specific knowledge for different degradations and shared knowledge for common patterns. However, existing methods: 1) rely on extra su-pervision signals, which are usually unknown in real-world applications; 2) employ fixed network structures, which re-strict the diversity of weather-specific knowledge. In this paper, we propose a Language-driven Restoration frame-work (LDR) to alleviate the aforementioned issues. First, we leverage the power of pre-trained vision-language (PVL) models to enrich the diversity of weather-specific knowl-edge by reasoning about the occurrence, type, and severity of degradation, generating description-based degradation priors. Then, with the guidance of degradation prior, we sparsely select restoration experts from a candidate list dy-namically based on a Mixture-of-Experts (MoE) structure. This enables us to adaptively learn the weather-specific and shared knowledge to handle various weather conditions (e.g., unknown or mixed weather). Experiments on exten-sive restoration scenarios show our superior performance.