Intuitive Robot Arm Control System Based on HMD Hand Tracking

Seung Jun Lee, Tae Hyeong Kim, Hyeong Jin Do, Kyudan Jung, Hyun Gon Ryu, Nam Joon Kim

Abstract: In this paper, we propose a system for intuitively controlling a robotic arm using hand tracking through a head-mounted display (HMD). It is necessary to transmit the angle values to each motor to control the robotic arm by tracking the human body. At this time, the system needs to know the extent to which each human joint is rotated, that is, the joint values. Based on these values, appropriate angles are sent to the motors. The joint values and positions of the user's hand were captured using the HMD, and the joint values for the arm and shoulder were calculated using inverse kinematics in Unity. A system is implemented to control the robotic arm by transmitting these values to the MCU. The tests confirmed that the system enables faster movements and better adaptability than traditional controllers, with a high accuracy of 98.5 %. In the future, we plan to enable long-distance remote control via an LTE module and display measurement values obtained from a measurement module on the HMD, enhancing convenience for the operator.