Transferring Movement Understanding for Parkinson’s Therapy by Generative Pre-Training

Published: 27 Oct 2023, Last Modified: 29 Jan 2024DGM4H NeurIPS 2023 PosterEveryoneRevisionsBibTeX
Keywords: language modeling, Parkinson's, motion capture, sequence modeling, GPT, scaling laws
TL;DR: Uniformly quantize motion to turn it into text and train language models on it for fine-tuning on clinical tasks
Abstract: Motion data is a modality of clinical importance for Parkinson's research but modeling it typically requires careful design of the machine learning system. Inspired by recent advances in autoregressive language modeling, we investigate the extent to which these modeling assumptions may be relaxed. We quantize motion capture data into discrete tokens and apply a generic autoregressive model to learn a model of human motion. Representing both positions and joint angles in a combined vocabulary, we model forward and inverse kinematics in addition to autoregressive prediction in 3D and angular space. This lets us pre-train on a 1B token, 40 hour dataset of motion capture, and then finetune on one hour of clinically relevant data in a downstream task. Despite the naivety of this approach, the model is able to perform clinical tasks and we demonstrate high performance classifying 5 hours of dance data.
Submission Number: 51