Mixed Reality Meets Robotic Systems: A Hololens2-Enabled Waypoint Navigation Interface
Keywords: Human-Robot Interaction
TL;DR: This paper presents a novel open-source Mixed Reality system that integrates the Microsoft HoloLens2, an overhead camera, and a ground rover to enable intuitive, holographic-based waypoint control for robotic navigation
Abstract: User interfaces for robotic systems often force users
to shift focus between digital interfaces and their physical
surroundings, which leads to inefficiencies and potential safety
issues. In this paper we present a novel Mixed Reality sys-
tem, which by seamlessly integrating holographic elements with
the physical world, seeks to overcome these limitations. Our
proposed system connects the Microsoft HoloLens2 with an
overhead camera and a ground rover to enable mixed reality-
based control for robotic navigation. The system allows users
to set waypoints for an autonomous rover using a holographic
interface displayed through the HoloLens2, providing an intuitive
and immersive control experience. The interface is engineered
with the objective of augmenting user awareness of both the
environmental context and system dynamics, and delivers real-
time visual feedback. With this proposed design, we address the
challenge of enhancing user multitasking and situational aware-
ness in complex environments. To the best of our knowledge,
this is the first open-source mixed reality human supervisory
control system supporting waypoint multi-robot control through
HoloLens2. The system has been tested by many users from our
department and demonstrated during educational and outreach
activities on campus (e.g., during lab tours). This paper discusses
the system’s design, implementation, and user experience, and
provides insights into future improvements and applications of
mixed reality in robotic control systems.
Submission Number: 5
Loading