GML-NeRF: Gate-guided Mutual Learning Framework for Neural Rendering

Lidong Guo, Xuefei Ning, Yonggan Fu, Tianchen Zhao, Zhuoliang Kang, Jincheng Yu, Yingyan Celine Lin, Yu Wang

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Neural rendering field, Mutual learning, Novel view synthesis, Soft gate module, Complex scenes with occlusions

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose a gate-guided multi-NeRF mutual learning framework applicable to various types of datasets.

Abstract:

Although the neural radiance field (NeRF) exhibits high-fidelity visualization on the rendering task, it still suffers from rendering defects in complex scenes. One of the primary reasons is the limited model capacity. However, directly increasing the network's width and depth cannot significantly improve the rendering quality. To address this issue, existing work adopts scene partitioning and assigns different 3D points to different network parameters. However, a 3D point may be invisible to some rays due to occlusions in complex scenes. On such a point, training with those rays that do not contain valid information about the point might interfere with the NeRF training. Based on the above intuition, we allocate model parameters in the ray dimension and propose a Gate-guided Mutual Learning framework for neural rendering (GML-NeRF). Specifically, we construct an ensemble of sub-NeRFs and train a soft gate module to assign the gating scores to these sub-NeRFs based on specific rays. The gate module is jointly optimized with the sub-NeRF ensemble, enabling it to learn the preference of sub-NeRFs for different rays automatically. Furthermore, we introduce depth-based mutual learning to enhance the rendering consistency among multiple sub-NeRFs and mitigate the depth ambiguity. Experiments on five diverse datasets demonstrate that GML-NeRF can enhance the rendering performance across a wide range of scene types compared with existing single-NeRF and multi-NeRF methods.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5430