Programming robots is a challenging task exacerbated by environmental factors, faulty hardware, and software bugs. When coding issues arise, traditional debugging techniques such as output logs or added print statements that may help in typical computer applications are not always useful for roboticists. This is because robots often have an array of sensors that output complex data types, which can be difficult to decipher as raw text data. As an alternative, we explore how an augmented reality head mounted display (ARHMD) may facilitate robotics programming by providing a medium for conveying 3D visualizations of robot data superimposed directly into the programmer’s environment. Such visualizations can provide users with an intuitive way to confirm their understanding of the robot’s inner state and sensor data. In this paper, we outline an augmented reality (AR) debugging tool for aiding roboticists with analyzing and fixing programs that elicit unwanted robot behavior and propose a series of planned studies to further understand the intersection of robotics programming and AR.

An AR Debugging Tool for Robotics Programmers

One focus of augmented reality (AR) in robotics has been on enriching the interface for human-robot interaction. While such an interface is often made intuitive to interact with,  it invariably imposes novel objects into the environment. In situations where the human already has a focus, such as in a human-robot teaming task, these objects can potentially overload our senses and lead to degraded teaming performance. In this paper, we propose using AR objects to solely augment natural objects to avoid disrupting our natural senses while adding critical information about the current situation. In particular, our case study focuses on addressing the limited field of view of humans by incorporating persistent virtual shadows of robots for maintaining situation awareness in proximal human-robot teaming tasks. 
We designed a novel process to generate such shadows and verified that they worked better than other commonly adopted methodologies in teaming scenarios.

Virtual Shadow Rendering for Maintaining Situation Awareness in Proximal Human-Robot Teaming

VR interfaces are promising for robotics for several reasons, including that they may be suitable for resolving many of the human performance issues associated with traditional robot teleoperation interfaces used for robot manipulation. In this systems-focused paper, we introduce and document the development of a VR-based robot control paradigm with manipulation assist control algorithm, which allows human operators to specify larger manipulation goals while leaving the low-level details of positioning, manipulation and grasping to the robot itself. For the community, we also describe system design challenges to our progress thus far.

Manipulation Assist for Teleoperation in VR

Virtual reality (VR) and social robotics have mutual benefits. VR offers an instrumented and manipulable environment in which robots and people can virtually interact as well as tools for visual manipulations of robot materiality and color. VR also has a wealth of knowledge about how multimodal communications like motion, proxemics, and touch can inform interaction. Submersing social robots in VR provides an opportunity for physically-grounded interaction that leverages behavioral anthropomorphism. This work attempts to intersect these previously disparate areas, eliciting participant storytelling about the simplest possible anthropomorphizable robot: a robot that approaches and then bumps into you. In the study, 16 participants experience twelve manifestations of virtual/physical robots that approach and collide into them. The moment of collision provides an opportunity for expressive interpretation that offers a first glimpse into future potentials for physically embodied companion characters in virtual reality.

Exploring Behavioral Anthropomorphism With Robots in Virtual Reality

With the rise of collaborative robots (cobots) in industrial workplaces, designing these robots to meet the needs of their human co-workers is becoming increasingly important. Psychological theories and recent results from HRI research suggest interpretability and predictability of cobot behavior as key factors for the establishment of trust and collaborative task success. However, it is not yet clear which robotic intention signals are easily interpretable for whom and how they might impact user experience in varying situations. Therefore, the transdisciplinary team of the research project CoBot Studio created an innovative virtual reality (VR) environment in which the effectiveness of different light- and motion-based cobot signals can be systematically evaluated (i) in various collaborative minigames, (ii) without safety concerns, (iii) under randomized controlled conditions, and (iiii) without the need to make actual hardware adjustments to a robot. This paper introduces functionalities and the system architecture of the highly versatile CoBot Studio research environment, presents an initial application, and discusses methodological benefits of using immersive VR games for research into human-robot collaboration.

CoBot Studio VR: A Virtual Reality Game Environment for Transdisciplinary Research on Interpretability and Trust in Human-Robot Collaboration 

Virtual reality (VR) interfaces for robots provide a three-dimensional (3D) view of the robot in its environment, which allows people to better plan complex robot movements in tight or cluttered spaces. In our prior work, we created a VR interface to allow for the teleoperation of a humanoid robot. As detailed in this paper, we have now focused on a human-in-the-loop planner where the operator can send higher level manipulation and navigation goals in VR through functional waypoints, visualize the results of a robot planner in the 3D virtual space, and then deny, alter or confirm the plan to send to the robot. In addition, we have adapted our interface to also work for a mobile manipulation robot in addition to the humanoid robot. For a video demonstration please see the accompanying video at https://youtu.be/wEHZug_fxrA.

Semi-Autonomous Planning and Visualization in Virtual Reality

Augmented Reality (AR) is a promising mode of communication for human-robot teaming, due in part to its ability to increase Situation Awareness. In this work, we specifically consider the role of AR in enabling Situation Awareness in time-dominant collaborative human-robot exploration tasks. We present a crowdsourced evaluation of a set of AR visualizations in a simulated version of such a task, and measure the Situation Awareness of experiment participants. Our results raise key questions about the efficacy of Situation Awareness measurement in observational (rather than interactive) evaluation contexts. 

You Have Time to Explore Over Here!: Augmented Reality for Enhanced Situation Awareness in Human-Robot Collaborative Exploration

This paper presents some early work and future plans regarding how the autonomous motions of a telepresence robot affect a person embodied in the robot through a head-mounted display. We consider the preferences, comfort, and the perceived naturalness of aspects of piecewise linear paths compared to the same aspects on a smooth path. In a user study, thirty-six subjects (eighteen females) watched panoramic videos of three different paths through a simulated museum in virtual reality and responded to questionnaires regarding each path. We found that comfort had a strong effect on path preference, and that the subjective feeling of naturalness also had a strong effect on path preference, even though people consider different things as natural. We describe a categorization of the responses regarding the naturalness of the robot's motion and provide a recommendation on how this can be applied more broadly. Although immersive robotic telepresence is increasingly being used for remote education, clinical care, and to assist people with disabilities or mobility complications, the full potential of this technology is limited by issues related to user experience. Our work addresses these shortcomings and will enable the future personalization of telepresence experiences for the improvement of overall remote communication and the enhancement of the feeling of presence in a remote location. 

Defining Preferred and Natural Robot Motions in Immersive Telepresence from a First-Person Perspective

Human teams exhibit both implicit and explicit intention sharing. To further development of human-robot collaboration, intention recognition is crucial on both sides. Present approaches rely on a vast sensor suite on and around the robot to achieve intention recognition. This relegates intuitive human-robot collaboration purely to such bulky systems, which are inadequate for large-scale, real-world scenarios due to their complexity and cost. In this paper we propose an intention recognition system that is based purely on a portable head-mounted display. In addition robot intention visualization is also supported. We present experiments to show the quality of our human goal estimation component and some basic interactions with an industrial robot. HAIR should raise the quality of interaction between robots and humans, instead of such interactions raising the hair on the necks of the human coworkers.    

HRI 2021I Workshop VAM-HRI Submissions