VmambaSCI: Dynamic Deep Unfolding Network with Mamba for Compressive Spectral Imaging
Abstract: Snapshot spectral compressive imaging can capture spectral information across multiple wavelengths in one imaging. The method, coded aperture snapshot spectral imaging (CASSI), aims to recover 3D spectral cubes from 2D measurements. Most existing methods employ deep unfolding framework based on Transformer, which alternately address a data subproblem and a prior subproblem. However, these frameworks lack flexibility regarding the sensing matrix and inter-stage interactions. In addition, the quadratic computational complexity of global Transformer and the restricted receptive field of local Transformer impact reconstruction efficiency and accuracy. In this paper, we propose a dynamic deep unfolding network with mamba for compressive spectral imaging, called VmambaSCI. We integrate spatial-spectral information of the sensing matrix into the data module and employs spatial adaptive operations in the stage interaction of the prior module. Furthermore, we develop a dual-domain scanning mamba (DSMamba), featuring a novel spatial-channel scanning method for enhanced efficiency and accuracy. To our knowledge, this is the first Mamba-based model for compressive spectral imaging. Experimental results on the public database demonstrate the superiority of the proposed VmambaSCI over the state-of-the-art approaches.
Primary Subject Area: [Experience] Multimedia Applications
Relevance To Conference: Hyperspectral images, compared to traditional RGB images, offer more bands (channels), enabling a more accurate and comprehensive depiction of captured scenes. In this way, by combining with other types of data in multimodal applications, the perception and quality of experience can be enhanced. In order to save time and effort to obtain hyperspectral images, the coded aperture snapshot spectral imaging (CASSI) system is designed. It scans the image scene in spatial channel dimension and captures it as a 2D measurement in the end. In our work, we aim to build a deep unfolding network with high accuracy, which reconstructs CASSI measurements back into the original 3D hyperspectral images. The idea of deep unfolding has caught attention in the community [1][2].
[1] Jiechong Song, Bin Chen, and Jian Zhang. 2021. Memory Augmented Deep Unfolding Network for Compressive Sensing. In Proceedings of the 29th ACM International Conference on Multimedia (MM '21). Association for Computing Machinery, New York, NY, USA, 4249-4258.
[2] Ping Wang and Xin Yuan. 2023. SAUNet: Spatial Attention Unfolding Network for Image Compressive Sensing. In Proceedings of the 31st ACM International Conference on Multimedia (MM '23). Association for Computing Machinery, New York, NY, USA, 5099-5108.
Submission Number: 1369
Loading