RNN-Based Visual Guidance for Enhanced Sense of Agency in Teleoperation With Time-Varying Delays

Tomoya Morita, Simon Armleder, Yaonan Zhu, Hiroto Iino, Tadayoshi Aoyama, Gordon Cheng, Yasuhisa Hasegawa

Abstract: Intuitive teleoperation enables operators to embody remote robots, providing the sensation that the robot is part of their own body during control. The sense of agency (SoA), i.e., the feeling of controlling the robot, contributes to enhanced motivation and embodiment during teleoperation. However, the SoA can be diminished by time-varying communication delays associated with teleoperation. We propose a visual guidance system to assist operations while maintaining a high SoA when teleoperating robots with time-varying delays, thereby improving positioning accuracy. In the proposed system, a recurrent neural network (RNN) model, trained on the pouring tasks of skilled operators, predicts the input position 500 ms ahead of the input from the novice operator and visually presents it in real-time as the end-effector target position. Experiments with time-varying delays confirmed that the proposed method provides a visual representation of the target position interpolated in time and space from the real-time input of the operator, guiding the operator to align with the trajectory of the skilled operator. The proposed method significantly improves task performance even under time-varying delays while maintaining a high SoA compared with other conditions. Applying the prediction system developed in this study to human-robot collaborative control may enable interventions that maintain the SoA.