REMSA: An LLM Agent for Foundation Model Selection in Remote Sensing

Binger Chen, Tacettin Emre Bök, Behnood Rasti, Volker Markl, Begüm Demir

Keywords: LLM Agent, Remote Sensing, Foundation Models, Benchmarking, Strcutured Database

Abstract: Foundation Models (FMs) are increasingly integrated into remote sensing (RS) pipelines for applications such as environmental monitoring, disaster assessment, and land-use mapping. These models include unimodal vision encoders trained in a single data modality and multimodal architectures trained in multiple sensor modalities, such as synthetic aperture radar (SAR), multispectral, and hyperspectral imagery, or jointly in image-text pairs in vision-language settings. FMs are adapted to diverse tasks, such as semantic segmentation, image classification, change detection, and visual question answering, depending on their pretraining objectives and architectural design. However, selecting the most suitable remote sensing foundation model (RSFM) for a specific task remains challenging due to scattered documentation, heterogeneous formats, and complex deployment constraints. To address this, we first introduce the RSFM Database (**RS-FMD**), the first structured and schema-guided resource covering over 150 RSFMs trained using various data modalities, associated with different spatial, spectral, and temporal resolutions, considering different learning paradigms. Built on top of RS-FMD, we further present **REMSA** (**Re**mote-sensing **M**odel **S**election **A**gent), the first LLM agent for automated RSFM selection from natural language queries. REMSA combines structured FM metadata retrieval with a task-driven agentic workflow. In detail, it interprets user input, clarifies missing constraints, ranks models via in-context learning, and provides transparent justifications. Our system supports various RS tasks and data modalities, enabling personalized, reproducible, and efficient FM selection. To evaluate REMSA, we introduce a benchmark of 75 expert-verified RS query scenarios, resulting in 900 task-system-model configurations under a novel expert-centered evaluation protocol. REMSA outperforms multiple baselines, including naive agent, dense retrieval, and unstructured retrieval augmented generation based LLMs, showing its utility in real decision-making applications. REMSA operates entirely on publicly available metadata of open source RSFMs, without accessing private or sensitive data. Our code and data will be publicly released.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 19780