From Pose to Muscle: Multimodal Learning for Piano Hand Muscle Electromyography

RUOFAN LIU, YICHEN PENG, Takanori Oku, Chen-Chieh Liao, Erwin Wu, Shinichi Furuya, Hideki Koike

Keywords: Multimodal Learning, Electromyography Estimation, Muscle Analysis

TL;DR: We introduce a multimodal electromyography (EMG) inference framework trained on a collected piano performance dataset, to enable high-fidelity hand EMG estimation and demonstrate generalization and adaptability across unseen users and tasks.

Abstract: Muscle coordination is fundamental when humans interact with the world. Reliable estimation of hand muscle engagement can serve as a source of internal feedback, supporting the development of embodied intelligence and the acquisition of dexterous skills. However, contemporary electromyography (EMG) sensing techniques either require prohibitively expensive devices or are constrained to gross motor movements, which inherently involve large muscles. On the other hand, EMGs exhibit dependency on individual anatomical variability and task-specific contexts, resulting in limited generalization. In this work, we preliminarily investigate the latent pose-EMG correspondence using a general EMG gesture dataset. We further introduce a multimodal dataset, PianoKPM Dataset, and a hand muscle estimation framework, PianoKPM Net, to facilitate high-fidelity EMG inference. Subsequently, our approach is compared against reproducible competitive baselines. The generalization and adaptation across unseen users and tasks are evaluated by quantifying the training set scale and the included data amount.

Primary Area: Applications (e.g., vision, language, speech and audio, Creative AI)

Submission Number: 13959