PhySpec: Physically Consistent Spectral Reconstruction via Orthogonal Subspace Decomposition and Self-Supervised Meta-Auxiliary Learning

Xingxing Yang, Jie Chen, Zaifeng Yang

TL;DR: A Physically Consistent Spectral Reconstruction Method via Orthogonal Subspace Decomposition and Self-Supervised Meta-Auxiliary Learnin

Abstract: This paper presents a novel approach to hyperspectral image (HSI) reconstruction from RGB images, addressing fundamental limitations in existing learning-based methods from a physical perspective. We discuss and aim to address the ``colorimetric dilemma": failure to consistently reproduce ground-truth RGB from predicted HSI, thereby compromising physical integrity and reliability in practical applications. To tackle this issue, we propose PhySpec, a physically consistent framework for robust HSI reconstruction. Our approach fundamentally exploits the intrinsic physical relationship between HSIs and corresponding RGBs by employing orthogonal subspace decomposition, which enables explicit estimation of camera spectral sensitivity (CSS). This ensures that our reconstructed spectra align with well-established physical principles, enhancing their reliability and fidelity. Moreover, to efficiently use internal information from test samples, we propose a self-supervised meta-auxiliary learning (MAXL) strategy that rapidly adapts the trained parameters to unseen samples using only a few gradient descent steps at test time, while simultaneously constraining the generated HSIs to accurately recover ground-truth RGB values. Thus, MAXL reinforces the physical integrity of the reconstruction process. Extensive qualitative and quantitative evaluations validate the efficacy of our proposed framework, showing superior performance compared to SOTA methods.

Lay Summary:

Problem:
Hyperspectral cameras capture rich light data but are costly. Converting standard RGB images to hyperspectral (HSI) via AI often fails to reproduce original colors accurately, creating unreliable results.

Solution:
We built PhySpec, an AI framework that combines physics principles and self-learning. It estimates camera color sensitivity and lighting effects to ensure HSI reconstructions match real-world physics. A novel self-training step (MAXL) fine-tunes the model for each new image using just a few adjustments, enforcing color accuracy.

Impact:
PhySpec produces more trustworthy hyperspectral data, crucial for medical imaging, environmental monitoring, and beyond. By grounding AI in physical rules, it advances reliable spectral analysis without expensive hardware.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Primary Area: Applications->Computer Vision

Keywords: Spectral reconstruction, meta-auxiliary learning, spectral sensitivity estimation, illumination estimation, subspace decomposition

Submission Number: 6627