CPEP: Contrastive Pose-EMG Pre-training Enhances Gesture Generalization on EMG signals

Wenhui Cui, Christopher Michael Sandino, Hadi Pouransari, Ran Liu, Juri Minxha, Ellen L. Zippi, Aman Verma, Anna Sedlackova, Erdrin Azemi, Behrooz Mahasseni

Keywords: EMG; Zero-shot Gesture Classification; Multimodal Pre-training

TL;DR: We propose a contrastive pre-training framework that enables zero-shot classification of unseen gestures from EMG signals.

Abstract: Hand gesture classification using high-quality structured data such as videos, images, and hand skeletons is a well-explored problem in computer vision. Leveraging low-power, cost-effective biosignals, e.g. surface electromyography (sEMG), allows for continuous gesture prediction on wearables. In this paper, we demonstrate that learning representations from weak-modality data that are aligned with those from structured, high-quality data can improve representation quality and enables zero-shot classification.Specifically, we propose a Contrastive Pose-EMG Pre-training (CPEP) framework to align EMG and pose representations, where we learn an EMG encoder that produces high-quality and pose-informative representations. We assess the gesture classification performance of our model through linear probing and zero-shot setups. Our model outperforms emg2pose benchmark models by up to 21% on in-distribution gesture classification and 72% on unseen (out-of-distribution) gesture classification.

Submission Number: 21