LUMEN: Language-guided Unified Memory for ENcoding Hyperspectral Images for Mars Mineral Identification

Abhiroop Chatterjee, Susmita Ghosh, Ashish Ghosh, Emmett Ientilucci

Keywords: hyperspectral imaging, few-shot learning, cross-modal alignment, semantic prototypes, memory-augmented encoder

TL;DR: We propose a hyperspectral visual encoder projecting into a shared space, stabilized with momentum memory, aligned with LLM-derived semantic prototypes for SOTA performance on Mars mineral identification tasks.

Abstract: Identifying Martian minerals from sparse hyperspectral observations is fundamentally constrained by extreme label scarcity and the absence of large-scale hyperspectral pretraining. To this end, we introduce LUMEN (Language-guided Unified Memory for ENcoding hyperspectral images), a memory-augmented cross-modal framework that formulates hyperspectral recognition tasks as structured relational alignment between spectral embeddings and language-derived semantic prototypes. LUMEN employs an ultralight 3D spectral encoder to distill spatial–spectral structure into a compact latent manifold, which is softly aligned to textual embeddings through a learnable bilinear projection. To reconcile linguistic priors with spectral evidence, semantic prototypes are refined via learnable offsets and a confidence-aware gating mechanism that adaptively regulates the contribution of language priors and visual memory. A unified momentum-updated memory bank further stabilizes learning by integrating exponentially smoothed visual representations with language-anchored prototypes, reducing cross-modal drift. Unlike prior language-guided or CLIP-style approaches, LUMEN explicitly enforces eigenspace-level cross-modal consistency through a spectral relational distillation objective that promotes global manifold alignment beyond pointwise embedding matching. Extensive evaluations of three Martian hyperspectral datasets show that LUMEN consistently exceeds strong transformer-based approaches while requiring fewer parameters and reduced training time.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 30