Semantic-aware Next-Best-View for Multi-DoFs Mobile System in Search-and-Acquisition based Visual Perception

Xiaotong Yu, Chang Wen Chen

Abstract: Efficient visual perception using mobile systems is crucial, particularly in unknown environments such as search and rescue operations, where swift and comprehensive perception of objects of interest is essential. In such real-world applications, objects of interest are often situated in complex environments, making the selection of the 'Next Best' view based solely on maximizing visibility gain suboptimal. Semantics, providing a higher-level interpretation of perception, should significantly contribute to the selection of the next viewpoint for various perception tasks. In this study, we formulate a novel information gain that integrates both visibility gain and semantic gain in a unified form to select the semantic-aware Next-Best-View. Additionally, we design an adaptive strategy with termination criterion to support a two-stage search-and-acquisition manoeuvre on multiple objects of interest aided by a multi-degree-of-freedoms (Multi-DoFs) mobile system. Several semantically relevant reconstruction metrics, including perspective directivity and region of interest (ROI)-to-full reconstruction volume ratio, are introduced to evaluate the performance of the proposed approach. Simulation experiments demonstrate the advantages of the proposed approach over existing methods, achieving improvements of up to 27.13\% for the ROI-to-full reconstruction volume ratio and a 0.88234 average perspective directivity. Furthermore, the planned motion trajectory exhibits better perceiving coverage toward the target.

Primary Subject Area: [Systems] Systems and Middleware

Relevance To Conference: This study contributes to multimedia and multimedia systems by proposing a semantic-aware Next-Best-View (NBV) scheme for efficient visual perception of the object of interest using mobile systems. By integrating both semantic gain and visibility gain, the proposed approach enhances the system's ability to selectively perceive the object of interest, leading to improved performance in tasks such as search and rescue operations. The adaptive strategy with termination criterion is developed for a two-stage search-and-acquisition manoeuvre, further balancing the focus on semantically-important objects while minimizing moving costs. The evaluation against different scenarios demonstrates significant improvements in reconstruction metrics, including ROI-to-full reconstruction volume ratio and perspective directivity. This advancement in the visual acquisition framework paves the way for more accurate and efficient scene perception, ultimately benefiting fields that rely on mobile platforms for complex tasks in unknown environments.

Submission Number: 2971