AFDFusion: An adaptive frequency decoupling fusion network for multi-modality image

Chengchao Wang, Zhengpeng Zhao, Qiuxia Yang, Rencan Nie, Jinde Cao, Yuanyuan Pu

Abstract:

The multi-modality image fusion goal is to create a single image that provides a comprehensive scene
description and conforms to visual perception by integrating complementary information about the merits
of the different modalities, e.g., salient intensities of infrared images and detail textures of visible images.
Although some works explore decoupled representations of multi-modality images, they struggle with complex
nonlinear relationships, fine modal decoupling, and noise handling. To cope with this issue, we propose an
adaptive frequency decoupling module to perceive the associative invariant and inherent specific among crossmodality by dynamically adjusting the learnable low frequency weight of the kernel. Specifically, we utilize
a contrastive learning loss for restricting the solution space of feature decoupling to learn representations
of both the invariant and specific in the multi-modality images. The underlying idea is that: in decoupling,
low frequency features, which are similar in the representation space, should be pulled closer to each other,
signifying the associative invariant, while high frequencies are pushed farther away, also indicating the intrinsic
specific. Additionally, a multi-stage training manner is introduced into our framework to achieve decoupling
and fusion. Stage I, MixEncoder and MixDecoder with the same architecture but different parameters are trained
to perform decoupling and reconstruction supervised by the contrastive self-supervised mechanism. Stage II,
two feature fusion modules are added to integrate the invariant and specific features and output the fused
image. Extensive experiments demonstrated the proposed method superiority over the state-of-the-art methods
in both qualitative and quantitative evaluation on two multi-modal image fusion tasks.