A Systematic Evaluation of Retrieval-Augmented Generation and Language Models for Space Operations

Ruben Catarino Belo, Marta Guimaraes, Claudia Soares

Keywords: Retrieval-Augmented Generation, Large Language Models, Space Operations Knowledge Management, Natural Language Processing for Space, Mission Operations Decision Support

TL;DR: This paper evaluates Retrieval-Augmented Generation and large language models for efficiently accessing and synthesizing domain-specific space operations documents to support safe, timely, and data-driven mission decisions.

Abstract:

The rapid expansion of space activities has led to an unprecedented accumulation of technical documentation, operational guidelines, and scientific literature, creating challenges for timely decision-making in space operations. Effective management in space operations requires tools capable of efficiently processing vast and heterogeneous information sources.

This paper systematically evaluates the performance of Retrieval-Augmented Generation (RAG) pipelines, combining Large Language Models (LLMs) with information retrieval techniques for extracting and synthesizing actionable knowledge from domain-specific documents. We compare various retrieval strategies, embedding models, and LLM answers to assess their impact on information accuracy, relevance, and reliability.

Our results demonstrate that RAG pipelines can significantly enhance knowledge access, reduce uncertainty, and support decision-making in complex space operations.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 23