Neural Spline Fields for Burst Image Fusion and Layer Separation

Ilya Chugunov, David Shustin, Ruyu Yan, Chenyang Lei, Felix Heide

Abstract: Each photo in an image burst can be considered a sam-ple of a complex 3D scene: the product of parallax, diffuse and specular materials, scene motion, and illuminant vari-ation. While decomposing all of these effects from a stack of misaligned images is a highly ill-conditioned task, the conventional align-and-merge burst pipeline takes the other extreme: blending them into a single image. In this work, we propose a versatile intermediate representation: a two-layer alpha-composited image plus flow model constructed with neural spline fields - networks trained to map input coordinates to spline control points. Our method is able to, during test-time optimization, jointly fuse a burst image capture into one high-resolution reconstruction and decom-pose it into transmission and obstruction layers. Then, by discarding the obstruction layer, we can perform a range of tasks including seeing through occlusions, reflection sup-pression, and shadow removal. Tested on complex in-the-wild captures we find that, with no post-processing steps or learned priors, our generalizable model is able to out-perform existing dedicated single-image and multi-view ob-struction removal approaches.