SceneAdapt: Scene-aware Adaptation of Human Motion Diffusion

Jungbin Cho, Minsu Kim, Jisoo Kim, Ce Zheng, Laszlo A. Jeni, Ming-Hsuan Yang, Youngjae Yu, Seon Joo Kim

Keywords: human motion generation, diffusion model adaptation, scene-aware human motion

TL;DR: We adapt human motion diffusion for scene-aware text conditioned generation with only scene-motion paired dataset.

Abstract: Human motion is inherently diverse and semantically rich, while also shaped by the surrounding scene. However, existing motion generation approaches fail to generate diverse motion while simultaneously respecting scene constraints, since constructing large-scale datasets with both rich text-motion coverage and precise scene interactions is extremely challenging. In this work, we introduce \textbf{SceneAdapt}, a framework that injects scene awareness into text-conditioned motion models by leveraging disjoint scene–motion and text–motion datasets through two adaptation stages: inbetweening and scene-aware inbetweening. The key idea is to use motion inbetweening, learnable without text, as a proxy task to bridge two distinct datasets, thereby injecting scene-awareness. In the first stage, we introduce keyframing layers that modulate motion latents for inbetweening while preserving the latent manifold. In the second stage, we add a scene-conditioning layer that injects scene geometry by adaptively querying local context through cross-attention. Experimental results show that \textbf{SceneAdapt} effectively injects scene awareness into text-to-motion models, and we further analyze the mechanisms through which this awareness emerges. Code and models will be released.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 21338