Multilayer Encoder-Decoder Network for 3D Nuclear Segmentation in Spheroid Models of Human Mammary Epithelial Cell Lines

Mina Khoshdeli, Garrett Winkelmaier, Bahram Parvin

Abstract: Nuclear segmentation is an important step in quantitative profiling of colony organization in 3D cell culture models. However, complexities arise from technical variations and biological heterogeneities. We proposed a new 3D segmentation model based on convolutional neural networks for 3D nuclear segmentation, which overcome the complexities associated with non-uniform staining, aberrations in cellular morphologies, and cells being in different states. The uniqueness of the method originates from (i) volumetric operations to capture all the three-dimensional features, and (ii) the encoder-decoder architecture, which enables segmentation of the spheroid models in one forward pass. The method is validated with four human mammary epithelial cell (HMEC) lines--each with a unique genetic makeup. The performance of the proposed method is compared with the previous methods and is shown that the deep learning model has a superior pixel-based segmentation, and an F1-score of 0.95 is reported.