% This is samplepaper.tex, a sample chapter demonstrating the
% LLNCS macro package for Springer Computer Science proceedings;
% Version 2.21 of 2022/01/12
%
\documentclass[runningheads]{llncs}
%
\usepackage[T1]{fontenc}
% T1 fonts will be used to generate the final print and online PDFs,
% so please use T1 fonts in your manuscript whenever possible.
% Other font encodings may result in incorrect characters.
%
\usepackage{graphicx}
\graphicspath{{./images}}

\usepackage{subcaption}

\usepackage{tabularx}
\usepackage{multirow}


% Used for displaying a sample figure. If possible, figure files should
% be included in EPS format.
%
% If you use the hyperref package, please uncomment the following two lines
% to display URLs in blue roman font according to Springer's eBook style:
\usepackage[pagebackref=true, breaklinks=true, colorlinks, bookmarks=false]{hyperref}
\usepackage{xcolor}
\renewcommand\UrlFont{\color{blue}\rmfamily}

%
%
\title{Semi-Supervised Abdomen Extraction and Organ Segmentation in CT Images}
%
\titlerunning{Semi-Supervised Abdomen Extraction and Organ Segmentation}
% If the paper title is too long for the running head, you can set
% an abbreviated paper title here
%

\newcommand*\samethanks[1][\value{footnote}]{\footnotemark[#1]}

\author{Rabin Adhikari\orcidID{0000-0001-5019-2205}\thanks{Equal Contribution} \and Safal Thapaliya\orcidID{0000-0002-4463-6700}\samethanks \and Bishesh Khanal\orcidID{0000-0002-2775-4748}}
%
\authorrunning{Rabin Adhikari, Safal Thapaliya, and Bishesh Khanal}
% First names are abbreviated in the running head.
% If there are more than two authors, 'et al.' is used.
%
\institute{NepAl Applied Mathematics and Informatics Institute for research (NAAMII) \email{\{rabin.adhikari,safal.thapaliya,bishesh.khanal\}@naamii.org.np}}

\renewcommand{\theenumi}{\alph{enumi}}

\begin{document}
%
\maketitle              % typeset the header of the contribution
%
\begin{abstract}
State-of-the-art supervised deep learning models perform well in segmenting abdominal organs from CT scan images when the test dataset has a similar distribution to the training dataset.
The lack of generalization of deep learning models means that one still needs to label new data and retrain the models for new datasets in a different environment.
However, expert annotation of multiple organs in volumetric scans is time-consuming and often prohibitively expensive.
Semi-supervised methods that can leverage unlabeled data together with few labeled data could be an attractive solution, but existing semi-supervised multiple organs semantic segmentation methods do not perform well.
Moreover, CT scans containing abdominal organs come with a great variety in the real clinical world with diverse resolutions and fields of view ranging from only the abdominal region to the whole body CT scan.  
In this paper, we propose a two-stage approach where abdomen region segmentation extracts the abdominal region, which is then fed as input to the abdominal organs segmentation network; both stages use the UNet architecture-based network.
To leverage unlabeled data, we use FixMatchSeg, which adapts the semi-supervised classification method, FixMatch, to segmentation tasks.
FixMatchSeg uses standard supervised loss with labeled examples and an unsupervised loss with pseudo labeling and consistency regularization that can leverage unlabeled samples.
%The unsupervised loss is designed with the idea that the model's prediction on weakly augmented and strongly augmented versions of the input image should be the same.
Our model improved the mean dice score from 31.54\% to 52.1\% on the validation set when utilizing 2,000 unlabeled training images over the 50 labeled images.
%was able to due to the distribution shift between labeled and unlabeled data.
%The model's performance could be considerably improved by investigating other ways to capitalize on the data distribution change.

\keywords{Semi-Supervised \and 3D U-Net \and Abdomen Region Segmentor \and Organ Segmentation \and Easy Implementation}
\end{abstract}

\section{Introduction}

Abdominal organ segmentation from CT scan images is an important medical imaging task with several clinical applications, such as aiding clinicians in diagnosing diseases, surgeons in therapy planning, and researchers in quantitative analysis for population-based studies.
In recent years, supervised deep learning segmentation methods have performed very well on curated datasets where the test sets do not have a significant distribution shift compared to the training set \cite{isensee2021nnu}.
However, one still needs to label a large number of data and train again, even for the same task, if the model is intended to be deployed for test data that has domain shift such as population demography, scanners, a field of view of input images, etc.
For multiple abdominal organs segmentation in 3D scans, manual annotation is tedious, time-consuming, and expensive.
Thus, semi-supervised methods that leverage several unlabeled samples together with a few labeled instances can be an attractive solution to develop powerful segmentation models for new datasets in new environments.
FLARE 2022 challenge\footnote{\url{https://flare22.grand-challenge.org/}} is focused on pushing the frontiers of semi-supervised segmentation methods.
Additionally, the challenge is interested in building models that are not too big and can run relatively fast with an economical computing budget.

There are various ways to leverage unlabeled samples in semi-supervised segmentation models such as consistency regularization methods, proxy-label or pseudo labeling methods, generative models, and graph-based techniques \cite{ouali2020overview}.
Among these approaches, consistency regularization and pseudo labeling are common, widely used, and have shown promising results in both natural images and medical images \cite{bai2017semi,baur2017semi,bortsova2019semi,cui2019semi,li2020transformation,li2020self}.
%generative modeling \cite{sedai2017semi,zhang2017deep,nie2018asdnet,chartsias2018factorised}, and graph-based approaches \cite{ganaye2018semi}, have been researched for medical image segmentation. 
A simple but effective semi-supervised classification approach, introduced in FixMatch \cite{sohn2020fixmatch}, combines consistency regularization with pseudo-labeling.
Upretee et al. \cite{upretee2022fixmatchseg} show with FixMatchSeg that FixMatch can be adapted to semantic segmentation tasks, providing promising results in medical images as well.
Hence, we use FixMatchSeg as the semi-supervised approach for segmenting the abdominal region and abdominal organs.

% Pseudo labeling and consistency regularization are a commonly used semi-supervised approach that has been used in medical image segmentation as well \cite{bai2017semi,baur2017semi,bortsova2019semi,cui2019semi,li2020transformation,nie2018asdnet,yu2019uncertainty}.
% We adapt this approach, from FixMatchSeg \cite{upretee2022fixmatchseg} to our 3D segmentation.
% The pseudo labels can be predicted from the unlabeled data using the network trained on the labeled ones, and a term can be introduced in the loss for the model to predict the pseudo labels for the unlabeled images.
% Bai et al. \cite{bai2017semi} introduced a semi-supervised strategy to the medical domain where they alternatively updated the network parameters and the predicted ground truths for the unlabeled examples (pseudo-labels).
% Nie et al. \cite{nie2018asdnet} introduced a fully convolutional confidence network for adversarially training the segmentation network.
% They then suggested a region-attention-based semi-supervised learning technique based on the confidence map from the confidence network.
% Another idea could be to use consistency regularization and make the model predict the same for a set of transformations.
% Bortsova et al. \cite{bortsova2019semi} did the same by ensuring equivariance to elastic deformations.
% They used a Siamese architecture with two identical branches that each get a different image transformation and a composite loss function with supervised segmentation loss and an unsupervised term that promotes segmentation consistency between the predictions of the two branches.
% The ground truth for the unlabeled images can be the pseudo labels obtained by repeatedly improving the neural network with a self-supervised task \cite{li2020self}.
% The suggested pseudo labels may be thought of as an approximate representation of the uncertainty estimation produced by combining many models with a substantial reduction in inference time.

The challenge provides 50 labeled and 2,000 unlabeled CT images.
The task is to segment 13 abdominal organs, and a separate blind test set that is not available to the participants is used for the evaluation.
One significant difficulty we saw in the dataset was the distribution shift due to the additional field of view in the unlabeled and validation set compared to that of the labeled set.
A majority of the labeled images contain only the abdominal region, but the unlabeled and validation sets have images with the coverage extending up to the whole body.
Similarly, the image size, spacing, and voxel anisotropy of the images varied very widely, as can be seen in \autoref{table:anisotropy}.

\begin{table}[h]
    \centering
    \caption{Min and Max (top \& bottom) image size in pixels and pixel spacing in mm for various image sets.
    The unlabeled and validation set images have remarkably wide variation compared to the training set images.}
    \label{table:anisotropy}
    \begin{tabular}{llll}
        \textbf{Images}& \textbf{Labeled} & \textbf{Unlabeled} & \textbf{Validation Set} \\
        \hline
        
        \multirow{2}{*}{Image Size (px)} & [512, 512, 71]& [512, 79, 24] & [512, 512, 82] \\
        & [512, 512, 113] & [796, 768, 1059] & [512, 512, 1338] \\
        \hline
        \multirow{2}{*}{Pixel Spacing (mm)} & [0.65, 0.65, 2.5] & [0.35, 0.35, 0.45] & [0.62, 0.62, 0.8] \\
        & [0.98, 0.98, 5.0] & [1.37, 3.0, 8.0] & [1.52, 1.52, 5.05] \\
        \hline
    \end{tabular}
\end{table}

We propose two key ideas to leverage the unlabeled data and tackle the data diversity and the domain shift.
To leverage the unlabeled data, we use FixMatchSeg \cite{upretee2022fixmatchseg}.
For handling the diversity in size and spacing, and the presence of non-abdominal regions in unlabeled and validation data, we propose a two-stage approach: input image fed to an abdomen region segmentation network and an organ segmentation network that takes the output region extracted by the region segmentation network as input.
Such a two-stage approach has been successfully used in pancreas segmentation from CT images \cite{zhu20183d} and in last year's winning FLARE submission \cite{zhang2021efficient}.

%This enables the predicted segmentation map to be detailed with less consumption of resources.

\section{Method}
%###########################
\subsection{Preprocessing}
All the input images first go through the following steps: 

\subsubsection{Set to RAI orientation:}
Since the majority of images were found to be in RAI orientation, we automatically set the orientation of all the images to RAI by extracting the metadata from the images and transforming them in the required direction using Monai\footnote{\url{https://monai.io/}} library.

\subsubsection{Resample input images:}
We resample the input images of dimensions $[x, y, z]$ to the target size of $[x', y', z'] = [128, \ge 128, \ge 64]$ in axes orthogonal to Sagittal, Coronal and Axial planes respectively.
Using the scaling factor $f = \frac{128}{x}$, resampled input images to the size $[128, \max(fy, 128), \max(fz, 64)]$ is fed to the Abdomen Region Segmentor.
The region extracted from the Abdomen Region Segmentor is then similarly resampled to the size of $[192, \ge 192, \ge 96]$ and fed to the Abdomen Organs Segmentor.

\subsubsection{Histogram Normalization:}
%We normalize the intensity of each data point using histogram normalization with 256 bins.
For each input sample, we perform histogram normalization followed by the intensity normalization to the range $[-1,\, 1]$ using \texttt{HistogramNormalize}\footnote{\url{https://docs.monai.io/en/stable/transforms.html\#histogramnormalize}} in Monai.
%It prevents the outliers in the intensity of the images dynamically.
%It may have resulted in a worse performance of the model due to the non-linear mapping of the intensities.
%It may become hard for the model to differentiate two organs based on their intensities due to the lower number of possible values (256 in our case).
%Future works may use clipping of the intensities beforehand and then converting them to z-scores ($\mu = 0$ and $\sigma = 1$).

\subsection{UNet based Semi-Supervised Abdominal Region and  Organs Segmentation}

\subsubsection{Two-stage segmentation:}

\begin{figure}[t]
    \includegraphics[width=\textwidth]{flare_model_final}
    \caption{The two-stage approach for abdominal organs segmentation.
    The Abdomen Region Segmentor (ARS) is first trained on a binary segmentation task where the foreground is the convex hull of all the abdominal organs.
    The abdomen region extracted from the ARS output is then fed as input to Abdomen Organs Segmentor after cropping and resizing.
    %Finally, the organs segmentor output is padded with the cropped-out region during region extraction to generate a final output segmentation mask of the same size as the original input image.
    }
    \label{fig:network}
\end{figure}

\autoref{fig:network} shows the pipeline of our two-stage approach to first segment the abdomen region and then abdominal organs.
The Abdomen Region Segmentor (ARS) is trained for a binary segmentation task where the foreground is obtained as the convex hull of all the target abdominal organs.
The Abdomen Region Segmentor first finds the largest connected component from the ARS foreground predictions, and then the smallest box that includes the largest foreground component is extracted as the abdomen region.
The abdomen region is then fed as input to Abdomen Organs Segmentor after cropping and resizing.

\subsubsection{UNet with Dice-CE compound loss:}
We use UNet~\cite{ronneberger2015u} with 
instance normalization \cite{ulyanov2016instance} from Monai\footnote{\url{https://docs.monai.io/en/stable/networks.html\#unet}} for both the networks.
We used anisotropic kernels of size (5, 5, 3) as the last axis has a lower resolution, hence the bigger physical receptive field can be achieved even with a lower kernel size.
%The weak augmentations (as described in \href{sec:training_protocols}{Training Protocols}) were applied to the then calculated convex hulls.
Since compound loss functions have proven to be robust in various medical image segmentation tasks \cite{ma2021loss}, we used the weighted sum of soft Dice loss \cite{milletari2016v} without background (weight 1) and cross-entropy loss (weight 0.25) for training both the models.
%The actual formula of the loss used is shown below.
%\[    Loss = DiceLoss + 0.25 \times CELoss \]

%Here, we used Cross Entropy loss as a guiding loss to help us reach the minima when dice loss has reached a plateau.
%Increasing the weightage to the CE loss increases the tendency of the model to predict foreground as CE doesn't work as well as dice loss on an imbalanced dataset.

\subsubsection{FixMatchSeg to leverage unlabeled images:}
We implemented FixMatchSeg \cite{upretee2022fixmatchseg} to leverage the unlabeled examples for both the segmentation networks.
The segmentation model's prediction on a weakly augmented version of an input image is used as ground truth (pseudo-label) and the loss is computed with the model's prediction on a strongly augmented version of the same image.
The input sample with pseudo-label is used in training only when the average of the pixel-wise max of the softmax prediction is higher than a certain threshold, that is only the samples where the model is confident enough.
The transformations for weak and strong augmentations and the hyperparameters including the confidence threshold for pseudo-label are described in \autoref{sec:implementation_details}.

\subsubsection{Improving inference speed and reducing resource consumption:}
We JIT-compile our models\footnote{\url{https://pytorch.org/docs/stable/generated/torch.jit.script.html}} to convert them to torchscripts.
TorchScript stores its definitions in a graph rather than dynamically.
It combines numerous operators (kernels) into a single one, making inference quicker.
Additionally, to make inference faster, we use \texttt{torch.jit.optimize\_for\_inference}\footnote{\url{https://pytorch.org/docs/stable/generated/torch.jit.optimize_for_inference.html}}.

\subsection{Post-processing}

We use the following post-processing steps in the provided order.
\begin{enumerate}
    \item We use a sliding window inference with 25\% overlap between consecutive windows.
    When combining the multiple predictions in the overlap regions, we weight the confidence of each prediction using a Gaussian kernel ($\sigma = 0.125$) with its mean placed at the window center.
    The assumption for using such a kernel is that the network makes better predictions for the organs in the central region of the input image.
    \item We use \texttt{KeepLargestConnectedComponent}\footnote{\url{https://docs.monai.io/en/stable/transforms.html\#keeplargestconnectedcomponent}} from Monai to keep the largest connected component with connectivity set to 1.
\end{enumerate}


\section{Experiments}
\subsection{Dataset and evaluation measures\protect\footnote{This section, as suggested in the challenge guideline, is copied from the template provided by the challenge organizers.}}

The FLARE 2022 dataset is curated from more than 20 medical groups under the license permission, including MSD \cite{simpson2019large}, KiTS \cite{heller2020international,heller2021state}, AbdomenCT-1K \cite{ma2021abdomenct}, and TCIA \cite{clark2013cancer}.
The training set includes 50 labeled CT scans with pancreas disease; 2000 unlabeled CT scans with liver, kidney, spleen, or pancreas diseases.
The validation set includes 50 CT scans with liver, kidney, spleen, or pancreas diseases.
The testing set includes 200 CT scans where 100 cases have liver, kidney, spleen, or pancreas diseases and the other 100 cases have uterine corpus endometrial, urothelial bladder, stomach, sarcomas, or ovarian diseases.
The metadata does not contain disease, demography, and center or scanner information.

The evaluation measures consist of two accuracy measures: Dice Similarity Coefficient (DSC) and Normalized Surface Dice (NSD), and three running efficiency measures: running time, area under GPU memory-time curve, and area under CPU utilization-time curve.

\subsection{Implementation details}
\label{sec:implementation_details}

\subsubsection{Environment settings:}
The development environments and requirements are presented in \autoref{table:env}.


\begin{table}[h]
    \caption{Development environments and requirements.}
    \label{table:env}
    \begin{tabularx}{\textwidth}{l|X}
        \hline
        Linux version  & Ubuntu 20.04.4 LTS \\
        \hline
        CPU & Intel\textsuperscript{\textregistered} Core\textsuperscript{\texttrademark} i9-9820X CPU @ 3.30GHz \\
        \hline
        RAM & $16\times 4$ GB; 3200 MT/s\\
        \hline
        GPU & NVIDIA GeForce GTX 1080 Ti 11GB and NVIDIA TITAN Xp 12 GB\\
        \hline
        CUDA version & 11.7\\
        \hline
        Programming language & Python 3.8.10\\ 
        \hline
        Deep learning framework & PyTorch (Torch 1.12, Monai 0.9.0) \\
        \hline
        Specific dependencies & pytorch\_lightning \\
        \hline
        Link to code & \url{https://github.com/naamiinepal/flare-2022} \\
        \hline
    \end{tabularx}
\end{table}

\subsubsection{Data Augmentation Strategy:}
FixMatchSeg needs weak and strong augmentation.
The transformations for weak and strong augmentation were selected from the transformations used by nnU-Net \cite{isensee2021nnu,isensee2019automated}.
The following augmentations, in order, were performed to the input images of the segmentor network.

\paragraph{Weak augmentation:}
\label{sec:weak_aug}
\begin{enumerate}
    \item Randomly rotate by $15^\circ$ in all directions with 0.9 probability with bilinear interpolation for image and nearest neighbor for the label with zero padding.
    \item Randomly zoom, with zoom value sampled from a Uniform distribution in the range (0.8, 1.3) with 0.9 probability, with trilinear interpolation.
    \item Randomly crop with size (128, 128, 64) for Abdomen Region Segmentor and (192, 192, 96) for Abdomen Organs Segmentor.
    If the input size for this step is less than the crop size, we zero pad instead of cropping.
\end{enumerate}

\paragraph{Strong augmentation, with a probability of 0.9:}
%, we do the operations that only change the image not affecting its segmentation map.
%The following was done on the image each with a probability of 0.9 in the provided order.
\begin{enumerate}
    \item Gaussian Random Noise with mean $\mu = 0$ and $\sigma = 0.1$
    \item Gaussian smoothing with $\sigma$ sampled from the Uniform distribution from the range (0.25, 1.5). 
    \item Multiplying the intensity by a factor sampled from the Uniform distribution from the range (0.7, 1.3).
    \item Gamma correction by a factor sampled from the Uniform distribution from the range (0.65, 1.5).
    \item Simulating low resolution by scaling down the image by a factor sampled from the Uniform distribution from the range (0.5, 1) by using the nearest neighbor interpolation and upscaling to the same shape using trilinear interpolation.
\end{enumerate}


\begin{table}[h]
    \centering
    \caption{Hyperparamenters for Abdomen Region Segmentor}
    \label{table:training}
    % \resizebox{0.47\textwidth}{!}{
    \begin{tabular}{l|l} 
        \hline
        Network initialization         & Kaiming Uniform \cite{he2015delving} \\
        \hline
        Batch size*                    & 4 \\
        \hline
        Gradient Accumulation Batches & 64 \\
        \hline 
        Patch size & $128\times128\times64$  \\ 
        \hline
        Total epochs & 100 \\
        \hline
        Optimizer          & Adam \cite{kingma2014adam} with betas=(0.9, 0.999)         \\ \hline
        Initial learning rate (LR)  & 0.01 \\ \hline
        Lr decay schedule & ReduceLROnPlateu with the patience of 3 epochs\\
        \hline
        Training time & 6.678 hours \\ 
        \hline
        Loss Function & Dice-CE Compound Loss \\
        \hline 
        Number of model parameters    & 637,425\\
        \hline
        Number of flops & 7.156G\\
        \hline
    \end{tabular}
\end{table}
% \footnotesize{* One patient's input is randomly cropped twice during the weak augmentation (see \href{sec:weak_aug}{Weak Augmentations}) to give two input samples to the Abdomen Region Segmentor which is sent in a single batch.}

\footnotesize{* One patient's input is randomly cropped twice during the weak augmentation (see \autoref{sec:weak_aug}) to give two input samples to the segmentors, which are sent in a single batch.}

\begin{table}[h]
    \centering
    \caption{Hyperparamenters for Abdomen Organs Segmentor}
    \label{table:training2nd}
    % \resizebox{0.47\textwidth}{!}{
    \begin{tabular}{l|l} 
        \hline
        Network initialization & Kaiming Uniform \cite{he2015delving} \\
        \hline
        Batch size* & 2 \\
        \hline 
        Gradient Accumulation Batches   & 64 \\
        \hline
        Patch size & $192\times192\times96$  \\ 
        \hline
        Total epochs & 50 \\
        \hline
        Optimizer          & Adam \cite{kingma2014adam} with betas=(0.9, 0.999) \\
        \hline
        Initial learning rate (LR)  & 0.05 \\
        \hline
        Lr decay schedule & ReduceLROnPlateu with the patience of 3 epochs\\
        \hline
        Training time & 19.02 hours \\ 
        \hline
        Loss Function & Dice-CE Compound Loss \\
        \hline 
        Number of model parameters    & 1.989M \\
        \hline
        Number of flops & 42.345G \\
        \hline
    \end{tabular}
\end{table}


\section{Results and discussion}

\subsection{Quantitative results on validation set}

\begin{table}[ht]
    \centering
    \caption{Dice Similarity Coefficient (DSC) comparison between training the model with only annotated images and training it using unlabeled images. Here, the model's mean DSC after using unlabeled images is higher than the DSC when using solely annotated images.}
    \label{table:compare_dsc}
    \begin{tabular}{lcc}
        % \cline{2-3}
        \textbf{Organ }& \textbf{Without Unlabeled Images} & \textbf{With Unlabeled Images} \\
        \hline
        % \hline
        Aorta & 0.6482 & 0.7786 \\
        % \hline
        Duodenum & 0.1459 & 0.3345 \\
        % \hline
        Esophagus & 0.3137 & 0.5225 \\
        % \hline
        Gallbladder & 0.2187 & 0.3603 \\
        % \hline
        Inferior Vena Cava & 0.5403 & 0.634 \\
        % \hline
        Left Adrenal Gland & 0.0400 & 0.3521 \\
        % \hline
        Left Kidney & 0.3381 & 0.5147 \\
        % \hline
        Liver & 0.5907 & 0.8453 \\
        % \hline
        Pancreas & 0.2032 & 0.4003 \\
        % \hline
        Right Adrenal Gland & 0.0644 & 0.3406 \\
        % \hline
        Right Kidney & 0.3334 & 0.493 \\
        % \hline
        Spleen & 0.5068 & 0.6993 \\
        % \hline
        Stomach & 0.1562 & 0.4978 \\
        Average & 0.3154 & 0.521\\
        \hline
    \end{tabular}
\end{table}

\autoref{table:compare_dsc} shows that our method leverages the unlabeled data, improving the mean dice score to 0.521 from 0.3154 when not using the unlabeled data.
However, the overall dice score is not high compared to the leaderboard which achieved 0.9064 on the validation set.
From our preliminary qualitative analysis, it seems that the abdomen localization could not be accurately done when the input images had whole-body CT scans or regions outside the abdomen.
In particular, a large majority of the labeled examples did not have a field of view outside the abdomen regions but the unlabeled and validation set had regions outside the abdomen.
It seems that the model was unable to train properly due to this domain shift.

\subsection{Qualitative results on validation set}
The segmentation findings for two successfully segmented images (case 21 and case 35) and two difficult-to-segment images (case 42 and case 44) are shown in \autoref{fig:well_segmented} and \autoref{fig:challenging}, respectively. 

Some images also have some organs missing (case 31 has the left kidney missing and case 44 has the right kidney missing).
Our model is not robust against such cases.

\begin{figure}[hp]
    \begin{subfigure}[t]{0.24\textwidth}
        \includegraphics[width=\textwidth]{21_coronal_img}
        \caption{a coronal slice; C21}
        \label{fig:21_coronal_img}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.24\textwidth}
        \includegraphics[width=\textwidth]{21_axial_img}
        \caption{an axial slice}
        \label{fig:21_img}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.24\textwidth}
         \includegraphics[width=\textwidth]{21_gt}
         \caption{ground truth}
         \label{fig:21_gt}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.24\textwidth}
         \includegraphics[width=\textwidth]{21_pred}
         \caption{model segmentation}
         \label{fig:21_seg}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.24\textwidth}
        \includegraphics[width=\textwidth]{35_coronal_img}
        \caption{a coronal slice; C35}
        \label{fig:35_coronal_img}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.24\textwidth}
        \includegraphics[width=\textwidth]{35_img}
        \caption{an axial slice}
        \label{fig:35_img}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.24\textwidth}
         \includegraphics[width=\textwidth]{35_gt}
         \caption{ground truth}
         \label{fig:35_gt}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.24\textwidth}
         \includegraphics[width=\textwidth]{35_pred}
         \caption{model segmentation}
         \label{fig:35_seg}
    \end{subfigure}
    \caption{The well-segmented images with mean DSC $0.71$(C21) and $0.70$(C35) are shown above.
    They have smaller file sizes and solely include CT scans of the abdomen.
    The model performs well in these circumstances because the annotated images include CT scans restricted to the abdomen areas.}
     \label{fig:well_segmented}
\end{figure}

\begin{figure}[hp]
    \begin{subfigure}[t]{0.24\textwidth}
        \includegraphics[width=\textwidth]{42_coronal_img}
        \caption{a coronal slice; C42}
        \label{fig:42_coronal_img}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.24\textwidth}
        \includegraphics[width=\textwidth]{42_img}
        \caption{an axial slice}
        \label{fig:42_img}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.24\textwidth}
         \includegraphics[width=\textwidth]{42_gt}
         \caption{ground truth}
         \label{fig:42_gt}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.24\textwidth}
         \includegraphics[width=\textwidth]{42_pred}
         \caption{model segmentation}
         \label{fig:42_seg}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.24\textwidth}
        \includegraphics[width=\textwidth]{44_coronal_img}
        \caption{a coronal slice; C44}
        \label{fig:44_coronal_img}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.24\textwidth}
        \includegraphics[width=\textwidth]{44_img}
        \caption{an axial slice}
        \label{fig:44_img}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.24\textwidth}
         \includegraphics[width=\textwidth]{44_gt}
         \caption{ground truth}
         \label{fig:44_gt}
    \end{subfigure}
    \hfill
    \begin{subfigure}[t]{0.24\textwidth}
         \includegraphics[width=\textwidth]{44_pred}
         \caption{model segmentation}
         \label{fig:44_seg}
    \end{subfigure}
    \caption{The challenging examples with mean DSC $0.28$(C42) and $0.23$(C44) are shown above.
    These CT scans cover a wider area of the body, such as the entire body. 
    Though in the case of patient 42, the image is restricted to the abdomen area, it contains a diseased right kidney whose size and intensity are unusual.}
    \label{fig:challenging}
\end{figure}

\subsection{Segmentation efficiency results}
The model validation was performed by running a docker container on a Ubuntu 20.04 desktop with the following specifications.

\begin{itemize}
    \item CPU: Intel\textsuperscript{\textregistered} Xeon(R) W-2133 CPU @ 3.60GHz $\times$ 12
    \item GPU: NVIDIA QUADRO RTX5000 (16G)
    \item RAM: 32G (Available memory 28G)
    \item Driver Version: 510.60.02   
    \item CUDA Version: 11.6
    \item Docker version 20.10.13
\end{itemize}

The average running time, GPU memory usage, AUC of the GPU time, and CPU time of the model evaluation are given in \autoref{table:efficiency}.
The running time, memory usage, and the AUC of both GPU and CPU curves were all observed on the 50 validation cases provided by the challenge's organizers.
In the example with a file size of 463 MB and more than 1000 slices (case 50), the maximum values of running time, GPU memory, Area under GPU memory-time curve, and Area under CPU utilization-time curve were found to be 34.02 seconds, 6.231 GB, 144.879 GBs, and 789.77 \%s, respectively.

\begin{table}[th]
    \caption{Average time of execution, GPU memory used, area under GPU memory-time curve, and area under CPU utilization-time curve for the 50 validation cases}
    \label{table:efficiency}
    \begin{tabularx}{\textwidth}{l|X|X|X}
        \hline
        \textbf{Running Time} & \textbf{Maximum GPU Memory} & \textbf{Area under GPU Memory-Time Curve} & \textbf{Area under CPU Utilization-Time Curve}\\
        \hline
        10.2456 s & 4.3975 GB & 27.275 GBs & 256.7952 \%s \\
        \hline
    \end{tabularx}
\end{table}

\subsection{Limitation and future work}
While the model seems to perform satisfactorily on input CT scans with the field of view restricted to the abdominal regions only.
The large majority of the labeled training images are of this type.
However, the unlabeled training images and the validation set images have domain shifts with a much larger field of view extending up to the whole body scan.
We believe that our method's pseudo-labeling approach could not provide good pseudo-labels on the unlabeled images with this domain shift when training the abdomen region segmenter.
Poor abdomen region segmentation then naturally leads to poor segmentation of the organs as the organs segmenter will not get the right input.
Similarly, our method does not address the missing organs and domain shift due to disease which makes it difficult to provide accurate segmentation of these cases.

A more detailed analysis of the failure cases and quantitative analysis such as the correlation of dice scores vs input field of view and abdomen region segmenter metrics will provide better insight into the issues with the current method.
Since the two-stage approach can be fragile when the first stage performs poorly, exploring end-to-end coarse-to-fine models with networks taking both coarse and fine resolution images together is another interesting direction.
Finally, semi-supervised methods other than FixMatchSeg and using self-supervision for pre-training can be explored.

\section{Conclusion}
FixMatchSeg was able to leverage the unlabeled data, improving the segmentation performance from 0.3154 mean DSC (only labeled data) to 0.521 DSC when adding the unlabeled data.
The final segmentation performance has reasonably low compared to the leaderboard toppers.
We believe this to be due to the inability of our method to handle the domain shift between the labeled and unlabeled images, and the validation-test images.
However, a more detailed error and failure analysis are needed in the future together with the exploration of other semi-supervised methods.

\subsubsection{Acknowledgments}
We declare that the segmentation method we implemented for participation in the FLARE 2022 challenge has not used any pre-trained models or additional datasets other than those provided by the organizers.
The proposed solution is fully automatic without any manual intervention.


%
% ---- Bibliography ----
%
% BibTeX users should specify bibliography style 'splncs04'.
% References will then be sorted and formatted in the correct style.
%
\bibliographystyle{splncs04}
\bibliography{ref}

\end{document}
