Using Language-Aligned Gesture Embeddings for Understanding Gestures Accompanying Math Terms

Tristan Maidment, Purav J. Patel, Erin Walker, Adriana Kovashka

Abstract: In this paper, we introduce an approach for recognizing and classifying gestures that accompany mathematical terms, in a new collection we name the "GAMT" dataset. Our method uses language as a means of providing context to classify gestures. Specifically, we use a CLIP-style framework to construct a shared embedding space for gestures and language, experimenting with various methods for encoding gestures within this space. We evaluate our method on our new dataset containing a wide array of gestures associated with mathematical terms. The shared embedding space leads to a substantial improvement in gesture classification. Furthermore, we identify an efficient model that excelled at classifying gestures from our unique dataset, thus contributing to the further development of gesture recognition in diverse interaction scenarios.

External IDs:dblp:conf/cvpr/MaidmentPWK22