Real-Time Fusion of Visual and Chart Data for Enhanced Maritime Vision

Marten Kreis, Dominik Hildebrand, Mahmut Karaaslan, Rafia Rahim, Andreas Zell, Ersin Kaya, Benjamin Kiefer

Keywords: Maritime Computer Vision, Multi-modal, Chart-Vision, Robotics

TL;DR: Associating chart data objects in image space works best with an end-to-end transformer

Abstract: This paper presents a novel approach to enhancing marine vision by fusing real-time visual data with chart information. Our system overlays nautical chart data onto live video feeds by accurately matching detected navigational aids, such as buoys, with their corresponding representations in chart data. To achieve robust association, we introduce a transformer-based end-to-end neural network that predicts bounding boxes and confidence scores for buoy queries, enabling the direct matching of image-domain detections with world-space chart markers. The proposed method is compared against baseline approaches, including a ray-casting model that estimates buoy positions via camera projection and a YOLO-based networks extended with a distance estimation module. Experimental results on a dataset of real-world maritime scenes demonstrate that our approach significantly improves object localization and association accuracy in dynamic and challenging environments.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 17