Semantic-Enhanced Digital Twin System for Robot-Environment Interaction Monitoring

Xin Li, Bin He, Zhipeng Wang, Yanmin Zhou, Gang Li, Rong Jiang

Abstract: In order to implement more accurate and fine-grained monitoring of robot-environment interaction (REI) in real time, this article extends the concept of digital twin (DT) to REI description and proposes a novel semantic-enhanced DT system with exteroceptive and proprioceptive modalities of robots as mechanisms for states monitoring and semantics inference associated with interactions. We create a 3-D generic graphic model embedded with contact dynamics based on domain descriptions and sensor data to mirror a real interactive scene and simultaneously extract the appearance and physical properties of real entities. In addition, we present a multifeature fused visual relations detector and introduce a lightweight spatial relation data set to measure the visual relationships between entity pairs. To keep the DT consistent with the response of a real system, the model is updated with interactive subprocess-triggered control. In our system, the whole framework is developed with virtual reality techniques, which provides a high degree of observability. The experiments on the objects sorting task in a kitchen scene are conducted, and the abilities for system monitoring in real time and semantic reasoning are evaluated; the results demonstrate the effectiveness and feasibility of our proposal.