Serial section microscopy image inpainting guided by axial optical flow

Yiran Cheng, Bintao He, Renmin Han, Fa Zhang

Abstract: Volume electron microscopy (vEM) is becoming a prominent technique in three-dimensional (3D) cellular visualization. vEM collects a series of two-dimensional (2D) images and reconstructs ultra-structures at the nanometer scale by rational axial interpolation between neighboring sections. However, section damage inevitably occurs in the sample preparation and imaging process, suffering from manual operational errors or occasional mechanical failures. The damaged regions present blurry and contaminated structure information, even local blank holes. Despite significant progress in single-image inpainting, it is still a great challenge to recover missing biological structures, that satisfy 3D structural continuity among sections. In this paper, we propose an optical flow-based serial section inpainting architecture to effectively combine the 3D structure information from neighboring sections and 2D image features from surrounding regions. We design a two-stage reference generation strategy to predict a rational and detailed intermediate state image from coarse to fine. Then, a GAN-based inpainting network is adopted to integrate all reference information and guide the restoration of missing structures, while ensuring consistent distribution of pixel values across the 2D image. Extensive experimental results well demonstrate the superiority of our method over existing inpainting tools.

Primary Subject Area: [Experience] Multimedia Applications

Secondary Subject Area: [Experience] Multimedia Applications, [Generation] Generative Multimedia

Relevance To Conference: In multimedia/multimodal processing, ensuring the integrity of serial image information is crucial for improving the quality of visual content. In this paper, we propose an optical flow-based vEM image inpainting framework called FlowInpaint to recover damaged regions in serial sections, considering both 3D structural continuity and 2D image content consistency. There are two main contributions in our work: First, we design a two-stage reference image generation strategy to estimate plausible biological structures in the recovered regions from coarse to fine. This strategy provides reliable cross-sectional reference which satisfies the 3D biological structural continuity for the image inpainting sub-network. Second, we adopt a GAN-based guided inpainting module to generate final seamless results. This module makes full use of multi-scale 2D image features from the reference information and ensures the structural consistency around the damaged regions. The comprehensive experiment results demonstrate the superiority of FlowInpaint in serial section inpainting over existing methods. This work enhances the availability of data, thereby increasing the number of training samples for the model. It helps to improve the accuracy and generalization ability of the model, and promotes the development of multimedia/multimodal processing.

Submission Number: 2257