Spectra without Spectra: Learning Spectral Reconstruction from Real-World RGB Data with Sparse Supervision

Shira Werman, Genady Paikin, Roy Shaul, Evgeny Soloveichik

Keywords: Spectral Reconstruction, Mixed Supervision, Weak Supervision, Deep Learning, Multispectral Imaging, Domain Gap, Real-World Data

TL;DR: We identify and bridge the synthetic-to-real gap in spectral reconstruction by training on real-world RGB images using sparse color-chart supervision and novel relighting consistency losses.

Abstract:

Our work addresses hyperspectral reconstruction from RGB images by identifying and mitigating a significant gap between synthetic and real-world data.
Methods mostly rely on synthetic RGB data generated from hyperspectral datasets, which we show generalizes poorly to real-world camera inputs.
To bridge this gap, we propose a novel training framework that leverages accessible real-world datasets containing RGB images with color charts and illumination measurements. Our mixed-supervision strategy, which is adaptable to existing state-of-the-art models, combines explicit supervision in regions with known spectra and physics-based self-regularization techniques across entire images.
In particular, sparse supervision is provided on color chart patches with measured spectral ground truth, serving as reliable anchors to guide learning across the full image.
We further strengthen reconstruction quality through two complementary regularizations to propagate spectral constraints across the entire image: a self-supervised RGB to RGB loss that enforces physically plausible reconstructions, and a relighting consistency loss that provides per-pixel target in the hyperspectral domain.
Experiments using the MST++ backbone on the BeyondRGB dataset demonstrate a substantial improvement in angular error (from 18.91° to 5.52°) compared to models trained on synthetic data.
This work offers a practical, accessible path toward robust real-world spectral reconstruction, moving beyond the reliance on synthetic data.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 1