TimeNeRF: Building Generalizable Neural Radiance Fields across Time from Few-Shot Input Views

Hsiang-Hui Hung, Huu-Phu Do, Yung-Hui Li, Ching-Chun Huang

Abstract: We present TimeNeRF, a generalizable neural rendering approach for rendering novel views at arbitrary viewpoints and at arbitrary times, even with few input views. For real-world applications, it is expensive to collect multiple views and inefficient to re-optimize for unseen scenes. Moreover, as the digital realm, particularly the metaverse, strives for increasingly immersive experiences, the ability to model 3D environments that naturally transition between day and night becomes paramount. While current techniques based on Neural Radiance Fields (NeRF) have shown remarkable proficiency in synthesizing novel views, the exploration of NeRF's potential for temporal 3D scene modeling remains limited, with no dedicated datasets available for this purpose. To this end, our approach harnesses the strengths of multi-view stereo, neural radiance fields, and disentanglement strategies across diverse datasets. This equips our model with the capability for generalizability in a few-shot setting, allows us to construct an implicit content radiance field for scene representation, and further enables the building of neural radiance fields at any arbitrary time. Finally, we synthesize novel views of that time via volume rendering. Experiments show that TimeNeRF can render novel views in a few-shot setting without per-scene optimization. Most notably, it excels in creating realistic novel views that transition smoothly across different times, adeptly capturing intricate natural scene changes from dawn to dusk.

Primary Subject Area: [Generation] Generative Multimedia

Relevance To Conference:

In the evolving landscape of multimedia processing, our work advances by enhancing the Neural Radiance Fields (NeRF) framework through TimeNeRF, which introduces novel modules capable of rendering dynamic scenes from limited input data across diverse viewpoints and temporal dimensions. The challenge of accurately translating visual scenes over time, ensuring both instance (temporal) accuracy and view (geometric) consistency, is particularly pronounced in real-world applications where data collection and optimization for variable viewpoints and times are complex and resource-intensive.
TimeNeRF addresses these challenges head-on, demonstrating unparalleled generalizability in a few-shot learning context and across arbitrary temporal changes. By enabling the creation of an implicit content radiance field for intricate scene representation and facilitating the generation of neural radiance fields tuned to specific times, TimeNeRF not only marks a significant leap in multimedia techniques but also pioneers a new research trajectory. It underscores the critical need for models capable of adapting flexibly to varied temporal and spatial viewing conditions, a necessity in the quest for more immersive and realistic multimedia experiences. Accordingly, the contribution of TimeNeRF aligns perfectly with the ACM Multimedia Conference’s mission to explore cutting-edge research and innovation in multimedia processing.

Submission Number: 3477