Design and Implementation of User Interface through Hand Movement Tracking and Gesture Recognition

Jeong Hoon Seong, Younggeun Choi

Abstract: In this paper, we investigate about how to track the movement of the hand and how to recognize the click gesture to implement a new type of user interface. We developed a wristwatch-type human-computer interface (HCI) device that can estimate and express the user's intuitive hand movements based on a 9-axis inertial measurement unit (IMU) sensor, which includes an accelerator, a magnetometer, and a gyroscope. We defined the Euler angular projection function to map the hand angle intuitively on the screen and to represent its motion reliably. We also proposed a machine-learning-based gesture-recognition algorithm by extracting the window size optimized for the click gesture and collecting the accurate ground truth in a real computing environment with noise. Finally, we designed a natural user interface, which is robust to the actual environment, by integrating hand motion tracking and click gesture recognition. We proved the reliability of motion by comparing the proposed hand motion-tracking function with the conventional method. In the experimental environment with noise, the click gesture recognition algorithm yielded a recognition rate of 98.94%. In conclusion, we modeled the optimized click gesture-recognition algorithm and integrated the mapping functions to track hand movements, and compared the system with existing interface devices. A usability test was performed for evaluation, and usability was verified compared with existing interface equipment.