Encoding Event-Based Gesture Data With a Hybrid SNN Guided Variational Auto-encoder

Kenneth Michael Stewart, Andreea Danielescu, Timothy Shea, Emre Neftci

Keywords: Neuromorphic Computing, Variational Auto-encoders, Representation Learning, Spiking Neural Networks, Self-supervised Learning

Abstract: Commercial mid-air gesture recognition systems have existed for at least a decade, but they have not become a widespread method of interacting with machines. These systems require rigid, dramatic gestures to be performed for accurate recognition that can be fatiguing and unnatural. To address this limitation, we propose a neuromorphic gesture analysis system which encodes event-based gesture data at high temporal resolution. Our novel approach consists of an event-based guided Variational Autoencoder (VAE) which encodes event-based data sensed by a Dynamic Vision Sensor (DVS) into a latent space representation suitable to compute the similarity of mid-air gesture data. We show that the Hybrid Guided-VAE achieves 87% classification accuracy on the DVSGesture dataset and it can encode the sparse, noisy inputs into an interpretable latent space representation, visualized through T-SNE plots. We also implement the encoder component of the model on neuromorphic hardware and discuss the potential for our algorithm to enable real-time, self-supervised learning of natural mid-air gestures.

One-sentence Summary: We present a novel algorithm for encoding and learning the latent representations of event-based gesture data with a proof of concept in Intel's Loihi neuromorphic hardware.