RelayGS: Reconstructing High-Fidelity Dynamic Scenes with Large-Scale and Complex Motions via Relay Gaussians

Qiankun Gao, Yanmin Wu, Chengxiang Wen, Jiarui Meng, Luyang Tang, Jie Chen, Ronggang Wang, Jian Zhang

Keywords: Gaussian Splatting, Dynamic Scene Reconstruction, 4D Reconstruction

Abstract: Reconstructing dynamic scenes with large-scale and complex motions—such as those in sports events—remains a significant challenge. Recent techniques like Neural Radiance Field and Gaussian Splatting have shown promise but often struggle with scenes involving substantial movement. In this paper, we propose **RelayGS**, a novel dynamic scene reconstruction method based on Gaussian Splatting, specifically designed to represent and learn large-scale complex motion patterns in highly dynamic scenes. Our RelayGS consists of three key stages. First, we learn the fundamental scene structure from all frames without considering temporal information and employ a learnable mask to decouple the highly dynamic foreground from the background exhibiting minimal motion. Second, we partition the scene into temporal segments, each consisting of several consecutive multi-view frames. For each segment, we replicate the foreground Gaussians, dubbed **Relay Gaussians**, as they are designed to act as relay nodes along the large-scale motion trajectory. By creating pseudo-views from frames uniformly selected from the segment, we optimize and densify foreground Relay Gaussians, further simplify and decompose large-scale motion trajectories into smaller, more manageable segments. Finally, we leverage HexPlane and lightweight MLPs to jointly learn the scene’s temporal motion field and refine the canonical Gaussians. We conduct extensive experiments on two dynamic scene datasets featuring large and complex motions to demonstrate the effectiveness of our RelayGS. RelayGS outperforms state-of-the-arts by more than 1 dB in PSNR, and successfully reconstructs real-world basketball game scenes in a much more complete and coherent manner, whereas previous methods usually struggle to capture the complex motion of players.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2805