Mobile AR-Based Robot Motion Control from Sparse Finger Joints

Di Wu, Shengzhe Chen, Meiheng Wang, Zhanglin Cheng

Abstract: Human-robot interaction plays a crucial role in the field of robot motion control. Recent advancements in Augmented Reality (AR) devices have opened up new possibilities for enhancing natural interaction. This paper explores the potential of utilizing AR devices as both sensors and interfaces for controlling robot motion. We present an intuitive and lightweight robot motion control system that enables users to manipulate a robot’s motion using hand gestures represented by 3D sparse finger joints. Our approach begins with the predefinition of primitive robot motions and their corresponding user gestures. Subsequently, we train a neural network to map 3D finger joints to specific robot motions. To enhance the accuracy of hand gesture recognition, we incorporate floating calibration and anchor-joint alignment techniques. An ablation experiment is devised to evaluate our proposed data alignment method, followed by a usability study conducted in real-world environments. Experimental results demonstrate the effectiveness of our approach in terms of accurate hand gesture recognition and intuitive control of robot motion.