CGRS: Continual Gesture Recognition System

Hun-Sung Park, Jaegyun Park, Dae-Won Kim, Jaesung Lee

Abstract: Gesture recognition using sensor data generated from mobile devices is used as a crucial part of human-computer interaction systems. These applications must allow users to easily add new gestures, and a continuous learning process is essential. However, retraining through the server is frequently required whenever a new gesture is added, resulting in significant power consumption. To address this problem, this paper presents the first on-device continual learning framework to continuously learn gestures on smartphones with limited resources. We introduce a continual gesture recognition system (CGRS) that uses replay-based continual learning. This approach retains previously learned information while simultaneously adapting to new gestures. We used replay buffer management technology to minimize memory usage on mobile devices with limited resources. CGRS achieved a high accuracy of 99.17% by evaluating continual learning performance as new gestures are added. Additionally, we successfully developed an application that can run on the Galaxy S10, a mobile device.