In recent years, neural network based image denoising approaches have revolutionized the analysis of biomedical microscopy data. Self-supervised methods, such as Noise2Void~(N2V), are applicable to virtually all noisy datasets, even without dedicated training data being available. Arguably, this facilitated the fast and widespread adoption of N2V throughout the life sciences. Unfortunately, we observed that the blind-spot training underlying N2V can lead to rather visible checkerboard artifacts, thereby reducing the quality of final predictions considerably. In this work, we present two modifications to the vanilla N2V setup that both help to reduce the unwanted artifacts considerably. Firstly, we propose a modified network architecture i.e. using blur-pool instead of max-pool layers throughout the used U-Net, rolling back the residual U-Net to a non-residual U-Net, and eliminating the skip connections at the uppermost U-Net level. Additionally, we propose new replacement strategies to determine the pixel intensity values that fill in the elected blind-spot pixels. We validate our modifications on a range of microscopy and natural image data. Based on added synthetic noise from  multiple noise types and at varying amplitudes, we show that both proposed modifications push the current state-of-the-art for fully self-supervised image denoising.

N2V2 - Fixing Noise2Void Checkerboard Artifacts with Modified Sampling Strategies and a Tweaked Network Architecture

Subcellular RNA localization is a critical mechanism for the spatial control of gene expression. Its mechanism and precise functional role is not yet very well understood. Single Molecule Fluorescence in Situ Hybridization (smFISH) images allow for the detection of individual RNA molecules with subcellular accuracy. In return, smFISH requires robust methods to quantify and classify RNA spatial distribution. Here, we present PointFISH, a novel computational approach for the recognition of RNA localization patterns. PointFISH is an attention-based network for computing continuous vector representations of RNA point clouds. Trained on simulations only, it can directly process extracted coordinates from experimental smFISH images. The resulting embedding allows scalable and flexible spatial transcriptomics analysis and matches performance of hand-crafted pipelines.

PointFISH: learning point cloud representations for RNA localization patterns

Computational methods for image-based profiling are under active development, but their success depends on assays that can maximize the phenotypic information captured. Fluorescent protein (FP) tags and other methods to fluorescently label proteins of interest provide a range of tools to investigate virtually any cellular process under the microscope. However, fluorescence microscopy is limited in the number of FPs that can be simultaneously imaged in the same cell. Cell segmentation methods often rely on the presence of morphological markers such as nuclei and cytoplasm. Here, we present a generalist approach to overcome the need for morphological reporters and that instead uses reporters that contain biological information for the segmentation of the nucleus and cytoplasm. Our method leverages state-of-the-art pre-trained segmentation models to segment cytoplasm and nuclei on fluorescence microscopy images. For this, we propose to fine-tune generalist networks for cell and nucleus segmentation for each individual fluorescent channel and to aggregate the respective segmentation results together. We assess our methodology performance and robustness across several fine-tuning strategies and fusion methods across different cell lines and various reporter proteins of a specific cellular signaling pathway, and for different acquisition methods. This approach will allow maximizing the extraction of relevant biological information to characterize cellular processes.

Cell Segmentation in Images Without Structural Fluorescent Reporters

In this paper we perform an empirical evaluation of variants of deep learning methods to automatically localize anatomical landmarks in bioimages of fishes acquired using different imaging modalities (microscopy and radiography). We compare two methodologies namely heatmap based regression and multivariate direct regression, and evaluate them in combination with several Convolutional Neural Network (CNN) architectures. Heatmap based regression approaches employ Gaussian or Exponential heatmap generation functions combined with CNNs to output the heatmaps corresponding to landmark locations whereas direct regression approaches output directly the $(x,y)$ coordinates corresponding to landmark locations. In our experiments, we use two microscopy datasets of Zebrafish and Medaka fish and one radiography dataset of gilthead Seabream. On our three datasets, the heatmap approach with Exponential function and U-Net architecture performs better. Datasets and open-source code for training and prediction will be published (upon paper acceptance) to ease future landmark detection research and bioimaging applications.


Empirical Evaluation of Deep Learning Approaches for Landmark Detection in Fish Bioimages

With the widespread creation of artificial intelligence (AI) models in biosciences, researchers are reusing AI models trained for specific tasks.
This work is motivated by the need to characterize AI models for reuse and dissemination based on metrics derived from optimization curves captured during model training. 
Such AI model characterization can aid future model accuracy refinement, inform users about model hyper-parameter sensitivity, and assist in model reuse according to multi-purpose objectives.
The challenges lie in understanding relationships between AI model characteristics and optimization curves, defining and validating quantitative AI model metrics,  and disseminating metrics with trained AI models. 
We approach these challenges by analyzing optimization curves generated for image segmentation and classification tasks with respect to AI model characteristics reused for many purposes.

Characterization of AI Model Configurations For Model Reuse

Attention Branch Networks (ABNs) have been shown to simultaneously provide visual explanation and improve the performance
of deep convolutional neural networks (CNNs). In this work, we introduce Multi-Scale Attention Branch Networks (MSABN), which enhance
the resolution of the generated attention maps, and improve the performance. We evaluate MSABN on benchmark image recognition and
fine-grained recognition datasets where we observe MSABN outperforms ABN and baseline models. We also introduce a new data augmentation strategy utilizing the attention maps to incorporate human knowledge in the form of bounding box annotations of the objects of interest. We show that even with a limited number of edited samples, a significant performance gain can be achieved with this strategy.

Towards Better Guided Attention and Human Knowledge Insertion in Deep Convolutional Neural Networks

Deep learning provides us with powerful methods to perform nucleus or cell segmentation with unprecedented quality. However, these methods usually require large training sets of manually annotated images, which are tedious --- and expensive --- to generate. In this paper we propose to use In Silico Labeling (ISL) as a pretraining scheme for segmentation tasks. The strategy is to acquire label-free microscopy images (such as bright-field or phase contrast) along fluorescently labeled images (such as DAPI or CellMask\texttrademark). We then train a model to predict the fluorescently labeled images from the label-free microscopy images. By comparing segmentation performance across several training set sizes, we show that such a scheme can dramatically reduce the number of required annotations.

Learning with minimal effort: leveraging ISL for segmentation

We present a method for deep neural network interpretability by combining feature attribution with counterfactual explanations to generate attribution maps that highlight the most discriminative features between classes. Crucially, this method can be used to quantitatively evaluate the performance of feature attribution methods in an objective manner, thus preventing potential observer bias. We evaluate the proposed method on six diverse datasets, and use it to discover so far unknown morphological features of synapses in Drosophila melanogaster. We show quantitatively and qualitatively that the highlighted features are substantially more discriminative than those extracted using conventional attribution methods and improve upon similar approaches for counterfactual explainability. We argue that the extracted explanations are better suited for understanding fine grained class differences as learned by a deep neural network, in particular for image domains where humans have little to no visual priors, such as biomedical datasets.

Discriminative Attribution from Paired Images

Progress in automated microscopy and quantitative image analysis has promoted high-content screening (HCS) as an efficient drug discovery and research tool. While HCS offers to quantify complex cellular phenotypes from images at high throughpout, this process can be obstructed by image aberrations such as out-of-focus image blur, fluorophore saturation, debris, a high level of noise, unexpected auto-fluorescence or empty images. While this issue has received moderate attention in the literature, overlooking these artefacts can seriously hamper downstream image processing tasks and hinder detection of subtle phenotypes. It is therefore of primary concern, and a prerequisite, to use HCS. In this work, we evaluate deep learning options that do not require extensive image annotations to provide a straightforward and easy to use semi-supervised learning solution to this issue. Concretely, we compared the efficacy of recent self-supervised and transfer learning approaches to provide a base encoder to a high throughpout artefact image detector. The results of this study suggest that transfer learning methods should be preferred for this task as they not only performed best here but present the advantage of not requiring sensitive hyperparameter settings nor extensive additional training.

Comparison of semi-supervised learning methods for High Content Screening quality control

The introduction of unsupervised methods in denoising has shown that unpaired noisy data can be used to train denoising networks, which can not only produce a high quality results but also enable us to sample multiple possible diverse denoising solutions. However, these systems rely on a probabilistic description of the imaging noise--a noise model. Until now, imaging noise has been modelled as pixel-independent in this context. While such models often capture shot noise and readout noise very well, they are unable to describe many of the complex patterns that occur in real life application. Here, we introduce an novel learning-based autoregressive noise model to describe imaging noise and show how it can enable unsupervised denoising for settings with complex structured noise patterns. We explore different ways to train a model for real life imaging noise and show that our deep autoregressive noise model has the potential to greatly improve denoising quality in structured noise datasets. We showcase the capability of our approach on various simulated datasets and on real photo-acoustic imaging data.

ECCV 2022 Workshop BIC Submissions