Physics-based Skinned Dance Generation with RL Fine-tuning

Jidong Jia, Youjian Zhang, Huan Fu, Dacheng Tao

Keywords: Dance generation, Reinforcement learning, Physical simulation

Abstract: Denoising Diffusion Probabilistic Models (DDPMs) have shown great potential in generating high-fidelity, diverse, natural dances consistent with given music. However, due to the scarcity of skinned human motion data and the complexity of mesh data, existing methods mainly focus on generating dance moves in the form of skeletons, overlooking the domain gap between the skeletal structure and the human body geometry. When skeletal motions are visualized with human body mesh, anomalies such as torso interpenetration and imbalanced movements become highly noticeable. This physical implausibility significantly diminishes the aesthetic appeal of the generated dances and hinders their practicality in real-world applications. To address this issue, we propose a physical reward to fine-tune the diffusion model. Specifically, We first train a motion imitation policy in a physical simulator and use it to evaluate the physical plausibility (e.g., penetration, foot sliding) of generated motions. Ideally, generated motions that are more physically plausible will be easier to imitate, which means higher rewards. So we fine-tune the diffusion model to generate more physically plausible motions through Reinforcement Learning Fine-Tuning (RLFT). Furthermore, we find that the physical reward tends to push the model to generate freezing motions for less torso intersections. To mitigate it, we proposed an anti-freezing reward to balance the preference for freezing motions. Experiments on the human dance dataset show that our method can significantly improve the physical plausibility of generated motions, thereby generating dances that are aesthetically pleasing and realistic.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2485