Deformable NeRF using Recursively Subdivided Tetrahedra

Zherui Qiu, Chenqu Ren, Kaiwen Song, Xiaoyi Zeng, Leyuan Yang, Juyong Zhang

Abstract: While neural radiance fields (NeRF) have shown promise in novel view synthesis, their implicit representation limits explicit control over object manipulation. Existing research has proposed the integration of explicit geometric proxies to enable deformation. However, these methods face two primary challenges: firstly, the time-consuming and computationally demanding tetrahedralization process; and secondly, handling complex or thin structures often leads to either excessive, storage-intensive tetrahedral meshes or poor-quality ones that impair deformation capabilities. To address these challenges, we propose DeformRF, a method that seamlessly integrates the manipulability of tetrahedral meshes with the high-quality rendering capabilities of feature grid representations. To avoid ill-shaped tetrahedra and tetrahedralization for each object, we propose a two-stage training strategy. Starting with an almost-regular tetrahedral grid, our model initially retains key tetrahedra surrounding the object and subsequently refines object details using finer-granularity mesh in the second stage. We also present the concept of recursively subdivided tetrahedra to create higher-resolution meshes implicitly. This enables multi-resolution encoding while only necessitating the storage of the coarse tetrahedral mesh generated in the first training stage. We conduct a comprehensive evaluation of our DeformRF on both synthetic and real-captured datasets. Both quantitative and qualitative results demonstrate the effectiveness of our method for novel view synthesis and deformation tasks. Project page: https://ustc3dv.github.io/DeformRF/

Primary Subject Area: [Experience] Multimedia Applications

Secondary Subject Area: [Experience] Interactions and Quality of Experience, [Content] Media Interpretation

Relevance To Conference:

This work significantly advances multimedia and multimodal processing by facilitating more realistic and dynamic creation of 3D content. The integration of tetrahedral meshes with feature grids in this research simplifies the process of deforming and animating complex objects, which is crucial for crafting lifelike scenes in various multimedia applications. This advancement enhances the realism and interactivity in areas like virtual reality, augmented reality, and interactive gaming, providing users with more immersive and engaging experiences.
Moreover, the innovative approach of iteratively computing barycentric coordinates addresses some of the computational challenges associated with high-resolution 3D models. This development is key for multimedia applications that handle complex 3D scenes, enhancing the feasibility of rendering detailed environments.
Overall, this research expands the capabilities for creating and manipulating 3D content within the realm of multimedia, offering richer, more detailed, and more engaging experiences. This can have a significant impact across various sectors, such as entertainment, education, and simulation, by enabling the creation of more captivating and nuanced multimedia content.

Submission Number: 2246