AstroCLIP: Cross-Modal Pre-Training for Astronomical Foundation Models

Published: 28 Oct 2023, Last Modified: 10 Dec 2023NeurIPS2023-AI4Science PosterEveryoneRevisionsBibTeX
Keywords: astrophysics, representation learning, contrastive learning
TL;DR: We build a cross-modal embedding that connects different kinds of astronomical observations of galaxies, allowing for retrieval by similarity and zero- and few- shot predictions.
Abstract: We present AstroCLiP, a strategy to facilitate the construction of astronomical foundation models that bridge the gap between diverse astronomical observational modalities. We demonstrate that a cross-modal contrastive learning approach between images and spectra of galaxies yields highly informative embeddings of both modalities. In particular, we apply our method on multi-band images and spectrograms from the Dark Energy Spectroscopic Instrument (DESI), and show that: (1) these embeddings are well-aligned between modalities and can be used for accurate cross-modal searches, and (2) these embeddings encode valuable physical information about the galaxies - in particular redshift and stellar mass - that can be used to achieve competitive zero- and few- shot predictions without further finetuning. Additionally, in the process of developing our approach, we also construct a novel, transformer-based model and pretraining approach for galaxy spectra.
Submission Track: Original Research
Submission Number: 150