Differentiable Rendering with Reparameterized Volume Sampling

Kirill Struminsky, Oleg Desheulin

Keywords: neural radiance fields, differentiable rendering, importance sampling, reparameterization trick

TL;DR: An importance sampling-based rendering algorithm for neural radiance fields based alleviates the costs of redundant radiance computation.

Abstract: We propose an alternative rendering algorithm for neural radiance fields based on importance sampling. In view synthesis, a neural radiance field approximates underlying density and radiance fields based on a sparse set of views of a scene. To generate a pixel of a novel view, it marches a ray through the pixel and computes a weighted sum of radiance emitted from a dense set of ray points. This rendering algorithm is fully differentiable and facilitates gradient-based optimization of the fields. However, in practice, only a tiny opaque portion of the ray contributes most of the radiance to the sum. Therefore, we can avoid computing radiance in the rest part. In this work, we use importance sampling to pick non-transparent points on the ray. Specifically, we generate samples according to the probability distribution induced by the density field. Our main contribution is the reparameterization of the sampling algorithm. It allows end-to-end learning with gradient descent as in the original rendering algorithm. With our approach, we can optimize a neural radiance field with just a few radiance field evaluations per ray. As a result, we alleviate the costs associated with the color component of the neural radiance field at the additional cost of the density sampling algorithm.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2302.10970/code)