Trinity: Unifying Class-Agnostic Terrain and Semantic Segmentation for Planetary Environments by Leveraging Synthetic Data

Marcus G Müller, Wout Boerdijk, Maximilian Durner, Abel Roman Gawel, Roland Siegwart, Rudolph Triebel

Keywords: semantic segmentation, terrain segmentation, open-set segmentation, planetary exploration, space perception

TL;DR: A transformer-based model called Trinity to jointly learn class-specific semantic and class-agnostic terrain segmentation by leveraging synthetic data to enable robot-independent terrain understanding

Abstract: Terrain understanding is fundamental for mobile robots operating in unstructured outdoor environments, in- cluding planetary surfaces on Earth, Mars, the Moon, and beyond. Existing vision-based traversability estimation methods rely on robot-specific annotations or semantic class mappings, limiting transferability across platforms and requiring costly re-annotation when robot capabilities change, while standard semantic segmentation methods only focus on specific prede- fined classes, which do not capture the variety of terrains. In this work, we propose a transformer-based architecture that jointly performs class-specific semantic segmentation and class-agnostic terrain segmentation within a unified network, called Trinity. Terrain regions are segmented based solely on visual appearance, without predefined semantic labels or robot- dependent traversability scores. This formulation enables the learning of robot-agnostic visual terrain priors that can be combined with robot-specific experience for downstream tasks such as traversability estimation, visual odometry, and mission planning. To enable large-scale training with diverse terrain appearances, we extend the OAISYS simulator and introduce RUGDSynth, a synthetic dataset inspired by RUGD with class- agnostic terrain samples. Furthermore, we present the EXTerra Dataset, providing real-world images annotated with both class- specific and class-agnostic terrain labels. Experiments demon- strate the feasibility of the proposed task and the effectiveness of our joint segmentation approach in complex planetary outdoor environment.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 9