A Sonar-Visual Dataset for Cross-Modal Underwater Robot Perception

Weitung Chen, Phil Tinn, Per Gunnar Auran, Martin Ludvigsen, Peter Halland Haro

Keywords: Underwater Robotics, Cross-Modal Learning, Dataset, Multibeam Sonar, Robotic Perception

TL;DR: Our paper introduces a sonar-visual dataset from ocean deployments, along with a cross-modal annotation tool and a proof-of-concept fish detection model, enabling cross-modal prediction research in underwater robotics.

Abstract: Underwater robots typically use both cameras and sonar for perception to leverage the rich semantic details of vision and the robust range measurements of acoustics. However, learning to map between these modalities via cross-modal prediction remains underexplored due to limited sonar-visual paired datasets. We present SOVIS, a sonar-visual dataset for cross-modal underwater perception. SOVIS comprises over 76,000 paired frames collected across 17 dives at six sites in the Trondheimfjord, supported by an end-to-end pipeline that cleans and synchronizes the cross-modal sensor data. We also introduce an interactive annotation tool designed to accelerate the labeling process for this paired data. Finally, we demonstrate a proof-of-concept cross-modal fish detection task using a small subset of labeled data, achieving a $7\times$ improvement in mAP@0.10 over a monocular camera baseline. SOVIS serves as the first step toward advancing cross-modal underwater perception research, enabling research directions such as dense sonar prediction from monocular images.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Paper Acceptance: No

Submission Number: 10