Programmable 3D snapshot microscopy with Fourier convolutional networks

Diptodip Deb, Zhenfei Jiao, Alex Bo-Yuan Chen, Misha Ahrens, Kaspar Podgorski, Srinivas C Turaga

Keywords: computational microscopy, computational photography, computer vision, deep learning

Abstract: 3D snapshot microscopy enables fast volumetric imaging by capturing a 3D volume in a single 2D camera image and performing computational reconstruction. Fast volumetric imaging has a variety of biological applications such as whole brain imaging of rapid neural activity in larval zebrafish. The optimal microscope design for this optical 3D-to-2D encoding is both sample- and task-dependent, with no general solution known. Deep learning based decoders can be combined with a differentiable simulation of an optical encoder for end-to-end optimization of both the deep learning decoder and optical encoder. This technique has been used to engineer local optical encoders for other problems such as depth estimation, 3D particle localization, and lensless photography. However, 3D snapshot microscopy is known to require a highly non-local optical encoder which existing UNet-based decoders are not able to engineer. We show that a neural network architecture based on global kernel Fourier convolutional neural networks can efficiently decode information from multiple depths in a volume, globally encoded across a 3D snapshot image. We show in simulation that our proposed networks succeed in engineering and reconstructing optical encoders for 3D snapshot microscopy where the existing state-of-the-art UNet architecture fails. We also show that our networks outperform the state-of-the-art learned reconstruction algorithms for a computational photography dataset collected on a prototype lensless camera which also uses a highly non-local optical encoding.

One-sentence Summary: We introduce a neural network architecture using efficiently implemented global kernels for 3D snapshot microscope optimization, volume reconstruction, and lensless computational photograph reconstruction.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2104.10611/code)