Alleviating tiling effect by random walk sliding window in high-resolution histological whole slide image synthesis

Shunxing Bao, Ho Hin Lee, Qi Yang, Lucas Walker Remedios, Ruining Deng, Can Cui, Leon Yichen Cai, Kaiwen Xu, Xin Yu, Sophie Chiron, Yike Li, Nathan Heath Patterson, Yaohong Wang, Jia Li, Qi Liu, Ken S. Lau, Joseph T. Roland, Lori A. Coburn, Keith T. Wilson, Bennett A. Landman et al. (1 additional authors not shown)

Keywords: Image synthesis, Tiling Effect, Inference

TL;DR: We present an unpaired synthesis model (MxIF to virtual H&E) that fully utilizes 27 MxIF channels and spatial constraints. A sliding window random shifting strategy is introduced during the optimized inference stage to alleviate the tiling effects.

Abstract: Multiplex immunofluorescence (MxIF) is an advanced molecular imaging technique that can simultaneously provide biologists with multiple (i.e., more than 20) molecular markers on a single histological tissue section. Unfortunately, due to imaging restrictions, the more routinely used hematoxylin and eosin (H&E) stain is typically unavailable with MxIF on the same tissue section. As biological H&E staining is not feasible, previous efforts have been made to obtain H&E whole slide image (WSI) from MxIF via deep learning empowered virtual staining. However, the tiling effect is a long-lasting problem in high-resolution WSI-wise synthesis. The MxIF to H&E synthesis is no exception. Limited by computational resources, the cross-stain image synthesis is typically performed at the patch-level. Thus, discontinuous intensities might be visually identified along with the patch boundaries assembling all individual patches back to a WSI. In this work, we propose a deep learning based unpaired high-resolution image synthesis method to obtain virtual H&E WSIs from MxIF WSIs (each with 27 markers/stains) with reduced tiling effects. Briefly, we first extend the CycleGAN framework by adding simultaneous nuclei and mucin segmentation supervision as spatial constraints. Then, we introduce a random sliding window shifting strategy during the optimized inference stage to alleviate the tiling effects. The validation results show that our spatially constrained synthesis method achieves a 56% performance gain for the downstream cell segmentation task. The proposed inference method reduces the tiling effects by using 50% fewer computation resources without compromising performance. The proposed random sliding window inference method is a plug-and-play module, which can be generalized and used for other high-resolution WSI image synthesis applications. The source code with our proposed model are available at https://github.com/MASILab/RandomWalkSlidingWindow.git