Learning Specific and General Realm Feature Representations for Image Fusion

Fan Zhao, Wenda Zhao

Abstract: A universal fusion framework for handling multi-realm image fusion reduces the cost of manual selection in varied applications. Addressing the generality of multiple realms and the sensitivity of specific realm, we propose a novel universal framework for multi-realm image fusion through learning realm-specific and realm-general feature representations. Shared principle network, adaptive realm feature extraction strategy and realm activation mechanism are designed for facilitating high generalization of across-realm and sensitivity of specific-realm simultaneously. In addition, we present realm-specific no-reference perceptual metric losses based on the edge details and contrast for optimizing the learning process, making the fused image exhibit more specific appearance. Moreover, we collect a new multi-realm image fusion dataset (MRIF), consisting of infrared and visual images, medical images and multispectral images, to facilitate our training and testing. Experimental results show that the fused image obtained by the proposed method achieves superior performance compared with the state-of-the-art methods on MRIF and the other three datasets including infrared and visual images, medical images and remote sensing images, respectively.