$X$-Planes: Adaptive and Efficient Representation for Dynamic Reconstruction and Rendering in the Age of Large Pretrained Models

Ding Yi, Hongfei Mao, Jianan Wang, Duomin Wang, Xianbiao Qi, Xiangyu Yue, Xili Dai

Keywords: Dynamic NeRF, Low rank and sparse decomposition, Large Reconstruction Model

TL;DR: We propose a dynamic NeRF pipeline that uses a Large Reconstruction Model for initialization and regularization, enabling state-of-the-art dynamic scene reconstruction through low-rank decomposition and feature distillation.

Abstract: In this paper, we present a novel dynamic NeRF pipeline with an effective initialization and distillation/optimization strategy. Previous approaches, such as \textit{K}-Plane and Tensor4D, rely on randomly initialized compact feature plane representations to model 4D dynamic scenes, grounded in tensor decomposition theory. In contrast, our method employs a pre-trained Large Reconstruction Model (LRM) to generate a noisy and incomplete initial 4D representation, subsequently factorizes it into compact feature planes via low-rank and sparse decomposition, and reuses the feature decoder of LRM to initialize the NeRF MLP. The decomposed feature planes and decoder serve as both an effective initialization and regularization for the dynamic NeRF optimization, enabling state-of-the-art results with enhanced performance. The pipeline is broadly applicable to dynamic NeRF methods, and readily benefits from future advancements of the LRM, paving the way for more generalizable dynamic NeRF tasks.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 24383