Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion Forecasting

Parth Kothari, Danya Li, Yuejiang Liu, Alexandre Alahi

Keywords: Motion Forecasting, Trajectory Prediction, Distribution Shifts, Transfer Learning

TL;DR: We propose a modular low-rank adaptation method that enables fast adaptation of deep motion forecasting models to new styles with limited samples

Abstract: Deep motion forecasting models have achieved great success when trained on a massive amount of data. Yet, they often perform poorly when training data is limited. To address this challenge, we propose a transfer learning approach for efficiently adapting pre-trained forecasting models to new domains, such as unseen agent types and scene contexts. Unlike the conventional fine-tuning approach that updates the whole encoder, our main idea is to reduce the amount of tunable parameters that can precisely account for the target domain-specific motion style. To this end, we introduce two components that exploit our prior knowledge of motion style shifts: (i) a low-rank motion style adapter that projects and adjusts the style features at a low-dimensional bottleneck; and (ii) a modular adapter strategy that disentangles the features of scene context and motion history to facilitate a fine-grained choice of adaptation layers. Through extensive experimentation, we show that our proposed adapter design, coined MoSA, outperforms prior methods on several forecasting benchmarks.

Student First Author: yes

Code: https://github.com/vita-epfl/motion-style-transfer

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2211.03165/code)