\documentclass{midl} % Include author names
%\documentclass[anon]{midl} % Anonymized submission

% The following packages will be automatically loaded:
% jmlr, amsmath, amssymb, natbib, graphicx, url, algorithm2e
% ifoddpage, relsize and probably more
% make sure they are installed with your latex distribution

\usepackage{mwe} % to get dummy images
\usepackage{wrapfig}
\usepackage{MnSymbol}
\usepackage{amsmath}
\usepackage{caption}

\makeatletter
\newcommand*\bigcdot{\boldsymbol{\cdot}}
\makeatother

% Header for extended abstracts
\jmlrproceedings{MIDL}{Medical Imaging with Deep Learning}
\jmlrpages{}
\jmlryear{2021}
% to be uncommented for submissions under review
\jmlrworkshop{Short Paper -- MIDL 2021 submission}
% \jmlrvolume{-- Under Review}
% \editors{Under Review for MIDL 2021\vspace{-0.02cm}}

%\title[Automatic PD-L1 positive cell detection]{Automated detection of PD-L1 positive tumor and
%immune cells in non-small cell lung cancer}

%\title{Multi-class point annotations for PD-L1 assessment}
%\title{Assessing PD-L1 in lung cancer with point annotations}
%\title{Assessing PD-L1 in lung cancer with a multi-class PseudoEdgeNet}
\title{$\mu$PEN: Multi-class PseudoEdgeNet for PD-L1 assessment}

 % Use \Name{Author Name} to specify the name.
 % If the surname contains spaces, enclose the surname
 % in braces, e.g. \Name{John {Smith Jones}} similarly
 % if the name has a "von" part, e.g \Name{Jane {de Winter}}.
 % If the first letter in the forenames is a diacritic
 % enclose the diacritic in braces, e.g. \Name{{\'E}louise Smith}

 % Two authors with the same address
 % \midlauthor{\Name{Author Name1} \Email{abc@sample.edu}\and
 %  \Name{Author Name2} \Email{xyz@sample.edu}\\
 %  \addr Address}

 % Three or more authors with the same address:
 % \midlauthor{\Name{Author Name1} \Email{an1@sample.edu}\\
 %  \Name{Author Name2} \Email{an2@sample.edu}\\
 %  \Name{Author Name3} \Email{an3@sample.edu}\\
 %  \addr Address}


% Authors with different addresses:
% \midlauthor{\Name{Author Name1} \Email{abc@sample.edu}\\
% \addr Address 1
% \AND
% \Name{Author Name2} \Email{xyz@sample.edu}\\
% \addr Address 2
% }

%\footnotetext[1]{Contributed equally}

% More complicate cases, e.g. with dual affiliations and joint authorship
\midlauthor{\Name{Jeroen Vermazeren\nametag{$^{1}$}} \Email{jeroen.vermazeren@radboudumc.nl}\\
\Name{Leander van Eekelen\nametag{$^{1}$}} \Email{leander.vaneekelen@radboudumc.nl}\\
\Name{Luca Meesters\nametag{$^{1}$}} \Email{Luca.Meesters@radboudumc.nl}\\
\Name{Monika Looijen-Salamon\nametag{$^{1}$}} \Email{Monika.Looijen-Salamon@radboudumc.nl}\\
\Name{Shoko Vos\nametag{$^{1}$}} \Email{shoko.vos@radboudumc.nl}\\
\Name{Enrico Munari\nametag{$^{2}$}} \Email{enrico.munari@unibs.it}\\
\Name{Caner Mercan\nametag{$^{1}$}} \Email{caner.mercan@radboudumc.nl}\\
\Name{Francesco Ciompi\nametag{$^{1}$}} \Email{francesco.ciompi@radboudumc.nl} \\
\addr $^{1}$ Radboud University Medical Center, Department of Pathology, Nijmegen, the Netherlands.\\
\addr $^{2}$ Department of Molecular and Translational Medicine, University of Brescia, Brescia,  Italy.\vspace{-0.1cm}
}

\begin{document}

\maketitle

\begin{abstract}
In this paper, we take the recently presented PseudoEdgeNet model to the level of multi-class cell segmentation in histopathology images solely trained with point annotations. We tailor its loss function to the challenges of multi-class segmentation and equip it with an additional false positive loss term. We evaluate it on the assessment of tumor and immune cells in PD-L1 stained lung cancer histopathology images, and compare it with YOLOv5.
\end{abstract}

\begin{keywords}
PseudoEdgeNet, YOLO, PD-L1, lung cancer, histopathology.\vspace{-0.02cm}
\end{keywords}

\setlength{\parindent}{0pt}

% = INTRODUCTION =
\section{Introduction\vspace{-0.05cm}}
Cell detection and segmentation in histopathology images are core steps in research and development of digital biomarkers that rely on counting, quantifying and analyzing shape and spatial interaction of multiple cell types. One example is the tumor proportion score (TPS), defined as the fraction of tumor cells positive to a PD-L1 immunohistochemical staining over all tumor cells in a tumor biopsy, assessed by pathologists to select non-small cell lung cancer (NSCLC) patients to receive immunotherapy. Computer assisted assessment of the TPS can address the current limitations in the subjective interpretation of PD-L1 expression at cell level, such as the presence of other PD-L1 positive cells (e.g. macrophages, to be excluded from the TPS) or the estimation uncertainty inherent in assessing potentially hundreds of thousands of cells in a tissue sample.
We propose to automate the cell quantification task at the core of patient selection by detecting, classifying and segmenting PD-L1 positive (\mbox{PD-L1$^+$}) and PD-L1 negative (\mbox{PD-L1$^-$}) tumor and immune cells in PD-L1 stained NSCLC tissue samples, a task that, to the best of our knowledge, has only been addressed in \cite{Alth19} using a closed-source solution. We developed \emph{$\mu$-PseudoEdgeNet} ($\mu$PEN), a novel multi-class formulation of PseudoEdgeNet (PEN) \cite{yoo2019pseudoedgenet}, to produce multi-class cell segmentation from cell point annotations. In addition to altering the loss terms for multi-class segmentation, we equip $\mu$PEN with a false-positive loss to promote specificity. We empirically show the contribution of each loss term and benchmark detection performance versus the state-of-the-art detection method YOLOv5\footnote{\href{https://doi.org/10.5281/zenodo.4418161}{doi:10.5281/zenodo.4418161}}.

% = METHOD = 
\section{$\mu$-PseudoEdgeNet}
$\mu$PEN updates and expands (Figure \ref{fig:uPEN}, green area) PEN's segmentation network $f$, edge network $g$ and attention network $h$, to predict multi-class cell segmentation when only trained with point annotations in input patches $I$ using the following loss function $L$:
\vspace{-0.6cm}

\small
\begin{equation*}
\hspace{-.4cm}
L(S,T) = - \underbrace{ 
        \frac{1}{\#P} \sum_{i \in P_{ }} \sum_{c>0} log(S_i^cT_i^c) - \frac{1}{\#V}\sum_{i \in V} log(S_{i}^{c=0})
    }_\text{\emph{Segmentation loss}} +  
    \underbrace{ 
        \vphantom{ \left(\frac{a^{0.3}}{b}\right) }  \lambda_1  \frac{1}{\#F_{ }} \sum_{i_{ }} |s(F) -  g(I)h(I)|
    }_\text{\emph{Edge loss}} -
    \underbrace{ 
        \vphantom{ \left(\frac{a^{0.3}}{b}\right) }  \lambda_2 \frac{1}{\#T^{c=0}} \sum_{i \in B_{fp}} log(S_{i}^{c=0})
    }_\text{\emph{False positive loss}},
\end{equation*} 
\normalsize
%
% == Figure: uPEN ==
\begin{wrapfigure}{r}{.47\linewidth}
\centering
\vspace{-0.5cm}
\includegraphics[width=\linewidth]{imgs/method11.pdf}
%\rule{0.9\linewidth}{0.75\linewidth}
\caption{$\mu$-PseudoEdgeNet.}
\label{fig:uPEN}\vspace{-.5cm}
\end{wrapfigure}
where $S_i^c$ is the softmax output at pixel $i$ and class $c$, $T_i^c$ is the point annotations in the ground truth matrix.
To compute the \emph{segmentation loss}, we first calculate the Voronoi boundaries $V$ from all point annotations $P$, from which we compute the cross-entropy with non-background predictions $S^{c>0}$. 
Similarly, we compute the cross-entropy with the background predictions $S^{c=0}$ for all pixels on Voronoi boundaries $V$.
To ensure cell boundary segmentation, the \emph{edge loss} is calculated as the sum of the absolute difference between the output of the Sobel filter $s$ and the element-wise multiplication of the output of the edge network $g$ and the attention module $h$, similar to the original PEN. 
In our multi-class version, we adapt the filter to be applied on the sum of foreground predictions, $F = \sum_{c>0}{S^c}$.
Inspired by \cite{laradji2018blobs}, we introduced a \emph{false positive loss} term by computing the cross-entropy of all connected components that cannot be associated with any point annotations in $P$. 
This loss term penalizes non-background predictions in false positive connected components denoted by $B_{fp}$, and we weigh its contribution by introducing the scaling constants $\lambda_{1}$ and $ \lambda_2$, which are set to 1 and 10 after empirical evaluation.
We train the same CNN backbones as in the original PEN from scratch with He initialization. At test time, we take the majority vote of all classes in a connected component and apply morphology based post-processing and test time color and shape augmentations. As a benchmark for $\mu$PEN, we also apply YOLOv5 to our dataset, using the default setting of its smallest model (YOLOv5s, 7.3M parameters, release 4.0), pretrained on the MS COCO dataset, only increasing the training IoU threshold to 0.25.

% = Experimental results =
\section{Experimental results}

% Material
We collected n=39 whole-slide NSCLC histopathology images from 33 patients, stained for PD-L1 and digitized at 40$\times$ magnification. A trained medical research assistant (LM) supervised by lung pathologists (MLS, SV) selected 87 regions of interest (ROI) of 250$\times$250 $\mu$m and manually annotated 32,180 cells in total with point annotations of tumor and immune cells, either PD-L1$^+$ or PD-L1$^-$ (four classes in total); annotations where checked by pathologists. We applied a data split of 21/9/9 for training/validation/testing, balanced at both patient and cell class level across sets. For YOLOv5, all point annotations were extended to bounding boxes of size 10$\times$10 $\mu$m.

% experiments
We trained $\mu$PEN and YOLOv5 on 512$\times$512px patches (0.25$\mu$m/px) and selected best models based on the lowest validation loss. We translated predictions to points by taking the center-of-mass of the segmentation ($\mu$PEN) or bounding box (YOLOv5) and compared models' performance via the F1 score: an annotation is a hit when a detection is within a 4$\mu$m distance (average radius of a cell in our dataset). Figure \ref{fig:results} depicts the incremental improvement of $\mu$PEN over PEN using the proposed false positive loss terms (0.611 vs. 0.653). It also shows that $\mu$PEN outperforms YOLOv5 when used ``off-the-shelf''. However, further hyper-parameter tuning could boost YOLOv5's performance.


\begin{figure}
\vspace{-.5cm}
\centering
\begin{minipage}{.4\textwidth}
  \centering
  \includegraphics[clip, trim=0.2cm .5cm .15cm .7cm, width=1\linewidth]{imgs/polar_chart(7).pdf}
  \label{fig:test1}
\end{minipage}%\
\hspace{0.2cm} \vspace{-0.2cm}
\begin{minipage}{.5\textwidth}
  \centering
  \includegraphics[clip, trim=0.2cm 13cm 13cm 0.3cm, width=1.0\linewidth]{imgs/visualization2(9).pdf}\vspace{-0.75cm}
  \label{fig:test2}
\end{minipage}
\captionof{figure}{(Left) Polar chart showing the F1 scores per class \& macro-averages. $\mu$PEN without FP loss is functionally equivalent to PEN. (Right) Visual results for YOLOv5/$\mu$PEN.}
\label{fig:results}
\end{figure}

\vspace{-0.2cm}

\section{Conclusions}
The output of $\mu$PEN can potentially power (semi-)automated TPS assessment via cell localization and classification, as well as future biomarker research based on spatial interaction, size and morphology of different cell types without the need to train with manual annotations of cell borders. This multi-class framework can be easily extended to include non-tumor and non-immune cells, making this approach applicable to whole-slide images.

\vspace{-0.2cm}

\bibliography{midl-samplebibliography}
\vspace{-0.8cm}

\end{document}

