Semi-Autonomous Planning and Visualization in Virtual RealityDownload PDF

Feb 15, 2021 (edited Mar 24, 2021)HRI 2021I Workshop VAM-HRI SubmissionReaders: Everyone
  • Keywords: Human-robot interaction (HRI), virtual reality (VR), shared control, manipulation planning, motion planning, Functional Waypoints
  • TL;DR: In this paper, we enable a user to teleoperate a robot from entirely within VR through new control methods and by providing human-in-the-loop planning through Functional Waypoints and visualizations of the robot's plan.
  • Abstract: Virtual reality (VR) interfaces for robots provide a three-dimensional (3D) view of the robot in its environment, which allows people to better plan complex robot movements in tight or cluttered spaces. In our prior work, we created a VR interface to allow for the teleoperation of a humanoid robot. As detailed in this paper, we have now focused on a human-in-the-loop planner where the operator can send higher level manipulation and navigation goals in VR through functional waypoints, visualize the results of a robot planner in the 3D virtual space, and then deny, alter or confirm the plan to send to the robot. In addition, we have adapted our interface to also work for a mobile manipulation robot in addition to the humanoid robot. For a video demonstration please see the accompanying video at
  • Track: full paper
  • Broader Impacts: Our work expands upon previous implementations of our Virtual Reality teleoperation interface with new control methods and visualizations to allow an operator to interact with the navigation and manipulation planners of a robot. This allows for human-in-the-loop planning in VR, where the operator can send goals via Functional Waypoints, then view and alter the plan proposed by the robot's planners.
2 Replies