\documentclass{midl} % Include author names
% \documentclass[anon]{midl} % Anonymized submission

% The following packages will be automatically loaded:
% jmlr, amsmath, amssymb, natbib, graphicx, url, algorithm2e
% ifoddpage, relsize and probably more
% make sure they are installed with your latex distribution
\usepackage{mathrsfs}
\usepackage{colortbl}
\usepackage{mwe} % to get dummy images
%\jmlrvolume{-- Accepted}
\jmlryear{2020}
\jmlrworkshop{Full Paper -- MIDL 2020}
%\editors{Accepted for MIDL 2020}

\title[addressing the mri reconstruction false negative problem]{Addressing The False Negative Problem of Deep Learning MRI Reconstruction Models by Adversarial Attacks and Robust Training}

% More complicate cases, e.g. with dual affiliations and joint authorship
\midlauthor{\Name{Kaiyang Cheng \midljointauthortext{Contributed equally} \nametag{$^{1,2}$}} \Email{victorcheng21@berkeley.edu} \AND
\Name{Francesco Caliv\'a \midlotherjointauthor \nametag{$^{2}$}} \Email{francesco.caliva@ucsf.edu} \AND
\Name{Rutwik Shah\nametag{$^{2}$}} \Email{rutwik.shah@ucsf.edu} \AND
\Name{Misung Han\nametag{$^{2}$}} \Email{misung.han@ucsf.edu} \AND
\Name{Sharmila Majumdar\nametag{$^{2}$}} \Email{sharmila.majumdar@ucsf.edu} \AND
\Name{Valentina Pedoia\nametag{$^{2}$}} \Email{valentina.pedoia@ucsf.edu}\\
\addr $^{1}$ Department of Electrical Engineering and Computer Sciences, University of California, Berkeley\\
\addr $^{2}$ $CI^{2}$, Center for Intelligent Imaging, Department of Radiology and Biomedical Imaging, University of California, San Francisco}

\begin{document}

\maketitle 

\begin{abstract}
Deep learning models have been shown to be successful in accelerating MRI reconstruction, over traditional methods. However, it has been observed that these methods tend to miss rare small features, such as meniscal tears, subchondral osteophyte, etc. in musculoskeletal applications. This is a concerning finding as these small and rare features are the particularly relevant in clinical diagnostic settings. Additionally, such potentially dangerous loss of details in the reconstructed images are not reflected by global image fidelity metrics such as mean-square error (MSE) and structural similarity metric (SSIM). In this work, we propose a framework to find the worst-case false negatives by adversarially attacking the trained models and improve the models'ability to reconstruct the small features by robust training.
\end{abstract}

\begin{keywords}
MRI Reconstruction, Adversarial Attack, Robust Training.
\end{keywords}

\section{Introduction}
High data quality is a priority in medical image analysis. Magnetic resonance imaging (MRI) has the capability of satisfying such requirement when it comes to screening soft tissues. Nevertheless, MRI has a limitation of requiring long scanning time. As a consequence, over the past few years, acceleration of MRI has received an increasing level of attention, which has not been restricted only to medical physicist but extended also to the deep learning community. The book chapter from \citet{hammernik2020machine}, the Fast-MRI challenge that was held at NeurIPS 2019~\cite{zbontar2018fastmri}, the AccelMR 2020 and MC-MRRec challenges at ISBI and MIDL 2020, in addition to dedicated sessions at international conferences such as ISMRM, MICCAI are all examples that deep learning-powered accelerated MRI is an active topic of research. 

\subsection{Hypotheses for false negative}
Despite a remarkable improvement in image quality of accelerated MRI from Deep Learning-based methods, the false negative reconstruction phenomenon is still present. 
%Despite the advances in accelerated MRI powered by Deep learning of the last few years, 
The false negative phenomenon of MRI reconstruction %models still presents, which 
refers to qualitative observations provided during the announcement of NeurIPS 2019 FastMRI challenge results\footnote{https://slideslive.com/38922093/medical-imaging-meets-neurips-4}. The top performing models in terms of structural similarity metric (SSIM) and radiologists' image quality assessment were shown to have failed in reconstructing some relatively small abnormalities such as meniscal tear and subchondral osteophyte. In the attempt to better explain this phenomenon we investigate two hypotheses:

\begin{itemize}
\item[1)] the information of small abnormality features is completely lost through the under-sampling process;
\item[2)] the information of small abnormality features is not completely lost. Instead, it is attenuated and laid in the tail-end of the distribution, hence is rare.
\end{itemize}
Were the first hypothesis true, it would be impossible for any method to reconstruct a small abnormality feature, unless the presence of the abnormality is confounded with other structural changes. We are unable to formally verify whether this hypothesis is always false. Nonetheless, we are able to demonstrate that the condition stated in hypothesis 1 is unlikely to occur.
Were the second hypothesis true, it would be possible for a reconstruction model to reconstruct it. This is especially true with data-driven and learning-based methods. In this work, we show that the second hypothesis is true in many cases, and it is possible for a deep learning reconstruction model to reconstruct the small abnormality by using the limited information available.

To investigate these two hypotheses, we define `\textit{false-negative adversarial feature}' (FNAF), a perceptible small feature which is present in the ground truth MRI but has disappeared upon MRI reconstruction, which is performed via a learning model. The contributions of this paper can be summarized as follows: 
\begin{itemize}
    \item [1)] We quantitatively show that highly performing deep learning reconstruction models trained to only maximize image quality can fail when it comes to reconstructing small and infrequent structures. %The transferability of false-negative adversarial feature between models suggests that FNAF are similar to the l-p norm bounded adversarial perturbation in the computer vision community. According to theoretical works \cite{bug_feature,adv_free}, adversarial examples are hard to avoid without having a defense mechanism in place. This means that without specific priors, it might be hard for a deep learning reconstruction model to reconstruct small and infrequent abnormalities, to be deployed in the actual clinical setting. We believe having a transferable, scalable (no labeling needed) and quantitative method for testing the clinical robustness of reconstruction models is as important as solving the false negative problem, as the community can use this as a benchmark for clinical relevance which goes beyond mere image quality in future works.
    \item [2)] We quantitatively show that it is possible to reconstruct small structures if the right set of priors is available during training, particularly if adversarial training is employed. %This corresponds to the second hypothesis in the paper. This is an important finding as we address the over-pessimism that no reconstruction models can reconstruct these features, as necessary information is lost when under-sampling.
\end{itemize}
\section{Related works}
\subsection{MRI Reconstruction with Deep Learning}
MRI reconstruction from undersampled k-space is key in fast MRI \cite{liang2019deep,hammernik2020machine}. \citet{liang2019deep} explains that deep learning-powered MRI reconstruction can be accomplished following either data-driven, model-driven or integrated approaches. Data-driven approaches are generally data hungry and do not require prior knowledge, mainly because they take advantage of a huge amount of data to learn the mapping between raw data and the reconstructed MRI. In model-based approaches, the solution space is restricted by injecting task prior knowledge. This can be obtained for instance by reproducing the iterative approach of compressed sensing. Integrated approaches combine positive aspects of both previous solutions.

\subsection{Adversarial attack by small perturbation}
% \subsection{Adversarial attacks}
To apply the practice of adversarial attack to %from the robustness literature in the area of 
MRI reconstruction with deep learning, it is important to understand the most studied forms of adversarial attack, adding small imperceptible perturbations to input images with the aim to mislead machine learning models
 \cite{biggio2013evasion,Szegedy2014,goodfellow}.   \citet{goodfellow, bubeck2018adversarialconstraints, gilmer2018adversarialsphere, mahloujifar2019curse, shafahi2018adversarialinevitable} attempts to develop a variety of theories that could explain these adversarial examples. One notable theory is that adversarial examples are a consequence of data scarcity \cite{more_data}, as the true data distribution is not being captured by non-sufficiently large dataset. 
Another profound explanation is provided by \citet{bug_feature}, which shows that adversarial successes are mainly supported by model's ability to generalize on standard test set by using non-robust features. In other words, adversarial examples are more likely a product of datasets rather than that of machine learning models. 
To make a model resistant to adversarial attacks without additional data, one could employ adversarial training and provide the model with a prior that remarks the fact that non-robust features are not useful \cite{goodfellow, madry2018towards}. These findings are orthogonal to the second investigated hypothesis: if we interpret the distribution of FNAF as the distribution of robust features, we may attribute FNAF reconstruction failure to the dataset's inability to capture FNAF's distribution. 
%
\subsection{Adversarial attack on generative networks}
 While most of adversarial attacks focus on discriminative models, \citet{generative} propose a framework to attack variational autoencoders (VAE) and the VAE-GAN. Specifically, input images are imperceptibly perturbed so that the generative models generate target images that belong to a different class. Although reconstruction models can be seen as generative, we differ from this work, mainly because we focus on generating perceptible features that perform un-targeted attacks.

\subsection{Adversarial attacks via bi- and three- dimensional transformations or physical attacks}
 Going beyond small perturbations, a set of more realistic attacks produced by 2D and 3D transformations has been proposed in \citet{xiao2018spatially,synthesizing}. Similarly to our work, these studies perform perceptible attacks. Arguably, the most realistic attacks are physical attacks, which are achieved by altering the physical space before an image is captured digitally \cite{physical}. \citet{Kgler2018PhysicalAI} propose a physical attack on Dermoscopy images by drawing on the skin, around areas of interest. Although these attacks could more easily translate to real world scenarios, it would be nearly impossible to perform physical attacks with imaging modalities such as MRI.\\

 Previously described adversarial attacks utilize the fact than certain small perturbations, spatial and textual transformation in the digital and/or physical world, do not alter the image semantics. This work utilize the fact that MRI reconstruction models should reconstruct all features of an under-sampled image. %Furthermore, the reconstructed image quality must match that of a fully sampled image or at least guarantee that the complete information is present.

\section{Methods}
\subsection{False negative adversarial attack on reconstruction networks}
Adversarial attacks aim to maximize the loss $\mathscr{L}$ of a machine learning model, parameterized by $\theta$. This can be achieved by changing a perturbation parameter $\delta$ within the set $S\subseteq R^d$ of the allowed perturbation distribution \cite{madry2018towards} -- which we restrict to be a set of visible small features in all the locations of an image. This can formally be expressed as:

% \citet{madry2018towards} formalize that the adversarial attack maximizes the loss of a machine learning model, parameterized by $\theta$. This is feasible by changing $\delta$ in the set of allowed distribution of $S \subseteq R^d$. This can be formalized as:

\begin{equation}\label{eq. standard_attack}
    \max_{\delta \in S} {\mathscr{L}}(\theta, x+\delta, y)
\end{equation}
$\mathscr{L}$ can be any arbitrary loss function. To apply \equationref{eq. standard_attack} to reconstruction, the reconstruction network aims to reconstruct all the features including the perturbation (small features). Conversely, the attacker aims to find the perturbation (small features) which the network is not capable of reconstructing. \\ 

Let $\delta$ be an under-sampled perturbation which is added to an undersampled image and $\delta^{\prime}$ the respective fully-sampled perturbation, the objective function becomes:
\begin{equation}\label{eq. fn_attack}
    \max_{\delta \in S} {\mathscr{L}}(\theta, x+\delta^{\prime}, y+\delta)
\end{equation}
with:
\begin{equation}\label{eq. delta_prime}
    \delta = U(\delta^{\prime})
\end{equation}
$U$ can be any under-sampling function, comprised of an indicator function $M$, which acts as a mask in the k-space domain, and an operator that allows for a conversion from image to k-space and vice-versa such as the Fast Fourier Transform (FFT) $\mathcal{F}$ and the inverse-FFT $\mathcal{F}^{-1}$. The under-sampling and the k-space mask $M$ functions are the same as the implementations provided by \citet{zbontar2018fastmri}.
\begin{equation}\label{eq. undersampling}
    U(y) = \mathcal{F}^{-1}(M(\mathcal{F}(y)))
\end{equation}
Since we synthetically construct the small added features, we can measure the loss value within the area occupied by each features and be aware if the features are reconstructed. In practice, we place a mask on the reconstructed image and the perturbed target image, so that only the area of the small feature is highlighted. The area is relaxed so that a small region at a distance $d$ from the feature border is also included. The motivation for the mask accounting for boundaries is: if only the loss of the FNAF’s foreground is measured, this might not capture some failure cases where the FNAF had blended-in with the background. Therefore, the loss is computed in a 5 pixels distance range from the boundary of the FNAF.
The loss is defined as
\begin{equation}\label{eq. attack_loss}
    \mathscr{L}=\alpha \cdot MSE(x,y) + \beta \cdot MSE(T(x),T(y)) 
\end{equation}
where x and y are the original and reconstructed MRIs respectively. T is an indicator function which masks over the FNAF in the ground truth and the reconstructed images. Weights $\alpha$ and $\beta$ are hyper-parameters set to 1 and 100 during adversarial training (described in details in section ~\ref{section:Training implementation details}) . This allows one to better preserve both image quality and robustness of FNAF. Conversely, during attack evaluation, $\alpha$ and $\beta$ are set to 0 and 1, to only evaluate FNAF reconstruction. The loss is maximized by either random search or finite-difference approximation gradient ascent.

\subsubsection{Random search}
We generate random shapes of feature $\delta$ at random locations in the image and find the $\delta$ that maximizes the loss in \equationref{eq. fn_attack}. Random search \cite{random_search} has been shown to be an effective optimization technique.

\subsubsection{Finite-difference approximated gradient ascent}
We notice that the location of the $\delta$ feature is an important factor in finding FNAF. To optimize for the low-dimensional non-differentiable parameter (\textit{i.e.} set of the ($x, y$) coordinates of $\delta$), we approximate the partial derivatives for each parameter $p$ with the finite central difference: 

\begin{equation}\label{eq. partialL}
    \frac{\partial L}{\partial p}= \frac{L\left(p+\frac{h}{2}\right)-L\left(p-\frac{h}{2}\right)}{h}
\end{equation} 
where $h$ is the step size. Gradient ascent is used to update the location parameter $p$ and maximize \equationref{eq. fn_attack}.

\subsection{Under-sampling information preservation verification}
A benefit of having a synthetic feature generator is that one can quantify the amount of preserved information after k-space under-sampling. To make sure the information of $\delta$ is preserved through under-sampling in the k-space, we make sure the following condition is fulfilled:
\begin{equation}\label{eq. epsilon}
    D(x+\delta, x) < \epsilon
\end{equation}
where $D$ is a distance function, and $\epsilon$ is a noise error tolerance threshold. We obtain $x+\delta^{\prime}$ and $x$ through the following: 
\begin{equation}\label{eq. Udelta_prime}
    U(y+\delta^{\prime}) = U(y)+U(\delta^{\prime}) \\ 
    = x+\delta
\end{equation} as $U$ is linear and closed under addition. MSE is used for $D$.

\subsection{FNAF-robust training}\label{section:FNAF-robust training}
Our attack formulation allows the reconstruction models to simultaneously undergo standard and adversarial training, while small perturbations-based adversarial training requires models to be trained only on robust features \cite{madry2018towards}. This allows one to do FNAF-robust training on a pre-trained model and speed up convergence. To accelerate training, we adopt ideas from \citet{adv_free}: in essence, to do FNAF-robust training, the model utilized a training set which included original and adversarial examples, including the examples that are generated during the search for the worst adversarial case. However, the inner maximization is performed by either random search or finite-difference approximation gradient ascent - described above. Random search reduces our implementation to be a data augmentation approach. Furthermore, strict adversarial training with random search in a worst-of-k fashion like in \cite{engstrom2017exploring} might result in improved model robustness, and its implementation in our framework is straightforward.

\section{Experiments and results}

\subsection{Experimental setup}\label{section:Experimental setup}
We conduct our experiments on the FastMRI knee dataset with single-coil setting, including 4x and 8x acceleration factors \cite{zbontar2018fastmri}. We evaluate our methods with two 2-D deep learning based methods, U-Net \cite{ronneberger2015u} -- an popupar baseline, and invertible Recurrent Inference Machines (I-RIM) \cite{putzky2019invert} -- the winner of the single-coil FastMRI challenge. For U-Net, we follow the training procedures described in \citet{zbontar2018fastmri}. For I-RIM we follow the training procedures described in \citet{putzky2019rim} and use the official released pre-trained model. 

\subsection{Implementation details}\label{section:Training implementation details}
We perform the FNAF attack on the models with a mean-square error (MSE) loss. We constrain the FNAF to comprise 10 connected pixels. The attack mask is placed within the center of a 120-by-120 crop of the image. The constraint ensures that the feature is small and placed in a reasonable location. For random search, 11 randomly shaped FNAF are generated at random locations for each sample in the validation set and the highest adversarial loss is recorded. For finite-difference gradient ascent (FD), we performed the optimizations for the location x and y in 2 iterations. 
The number of iterations is chosen to have a reasonable computation time and keep the number of forward passes for one sample constant for both methods. The FD step size $h$ is set to $10$ and the learning rate to $10^5$.

An attack is rejected when the information-preservation (IP) loss is lower than 0.0001. This is especially important for FNAF-robust training, as we do not want the FNAF-robust model to go to the other extreme and produce hallucination of non-existing features.
With regard to FNAF-robust training, the data augmentation approach training procedure described in Section~\ref{section:Training implementation details} is followed. The adversarial loss of \equationref{eq. attack_loss} is used with $beta$ set to 100 to force the model to focus on the small features. To prevent from overfitting in terms of FNAF attack successes, the best model in terms of the standard reconstruction loss on the validation set is selected to be attacked, ignoring the adversarial loss.

\subsection{Attack evaluation metrics}
The average attack loss for the validation set and the attack hit rate are calculated. The average attack loss is defined in \equationref{eq. attack_loss}. An attack is considered to be a hit when the loss is higher than a threshold value $\gamma$. We empirically set $\gamma$ to 0.001, as we observed the FNAF to be mostly lost when the loss is greater than 0.001. The hit rates are conservatively low, as $\gamma$ is set at a high value, so that there might be cases where the FNAF is lost even at loss values below $\gamma$. We speculate that the actual hit rate is likely higher than the value reported in this work. 

\subsection{Attack results}
Examples of the FNAF are shown in \figureref{fig:FNAF_attack}. The result of the attack shown in \tableref{tab:attack_models} confirms that hypothesis 2 is true in many cases. The attack with FD is weaker than that with random search (RS), which is counter-intuitive. This might be due to various reasons, such as tuning the optimizer hyper-parameters, the number of iterations, etc. Nonetheless, the high success rate of the random search method for both models show that it is fairly easy to find a FNAF in the search space that is heuristically defined. Although I-RIM is more resilient to the attacks than U-Net, the attack rate is still fairly high. This is concerning but also understandable given that deep learning methods are not explicitly optimized for such objective, so these FNAF are at the tail-end of the distribution or even out-of-distribution with respect to the training distribution. Fortunately, we can modify the objective as specified in Section~\ref{section:FNAF-robust training} to produce a FNAF-robust model which is empirically fairly resilient to the attacks and also has minimal effect in the standard reconstruction quality shown in \tableref{tab:standard_eval}.

\begin{table}[ht]
\textit{ % The first argument is the label.
 % The caption goes in the second argument, and the table contents
 % go in the third argument.
}\floatconts
  {tab:standard_eval}%
  {\caption{Standard validation set evaluation with SSIM and normalized mean-square error (NMSE)}}%
  {\begin{tabular}{|c|c|c|}
\hline
       4$\times$  & SSIM & NMSE                              \\ \hline
U-Net             & $0.7213\pm0.2621$ & $0.03455\pm0.05011$  \\ \hline
I-RIM             & $0.7501\pm0.2546$ & $0.03413\pm0.05800$  \\ \hline
FNAF-robust U-Net & $0.7197\pm0.2613$ & $0.03489\pm0.05008$  \\ \hline
\multicolumn{1}{c}{}\\
\hline
      8$\times$   & SSIM & NMSE                             \\ \hline
U-Net             & $0.6548\pm0.2942$ & $0.04935\pm0.04962$  \\ \hline
I-RIM             & $0.6916\pm0.2941$ & $0.04438\pm0.06830$  \\ \hline
FNAF-robust U-Net & $0.6533\pm0.2924$ & $0.04962\pm0.05670$  \\ \hline
\end{tabular}}
\end{table}

\begin{table}[ht]
 % The first argument is the label.
 % The caption goes in the second argument, and the table contents
 % go in the third argument.
\floatconts
  {tab:attack_models}%
  {\caption{FNAF attack evaluations.}}%
  {\resizebox{\columnwidth}{!}{\begin{tabular}{|c|c|c|c|c|}
\hline
       $4\times$           & RS (Attack Rate \%) & FD (Attack Rate \%) & RS (MSE) & FD (MSE) \\ \hline
U-Net             &      $84.44$          &      $72.17 $          &0.001530  & 0.001386\ \\ \hline
I-RIM             &      $44.49$          &      $34.60 $          &0.001164  & 0.001080 \\ \hline
FNAF-robust U-Net &      $12.71$          &      $10.48 $          &0.000483  &0.000466  \\ \hline

\multicolumn{1}{c}{}\\

\hline
       $8\times$           & RS (Attack Rate \%) & FD (Attack Rate \%) & RS (MSE) & FD (MSE) \\ \hline
U-Net             &        $86.00$        &       $74.84 $         & $0.001592$ &0.001457  \\ \hline
I-RIM             &        $77.39$        &       $63.88 $         & $0.001470$ &0.001349  \\ \hline
FNAF-robust U-Net &        $15.09$        &       $13.30 $         & $0.000534$ &0.000467  \\ \hline
\end{tabular}}}
\end{table}


\begin{figure}[ht]
 % Caption and label go in the first argument and the figure contents
 % go in the second argument
\floatconts
  {fig:FNAF_attack}
  {\caption{The top row (A-D) shows a "failed" FNAF attack. The bottom row (E-H) shows a "successful" FNAF attack. Column 1 contains the under-sampled zero-filled images. Column 2 contains the fully-sampled ground truth images. Column 3 contains U-Net reconstructed images. Column 4 contains FNAF-robust U-Net reconstructed images. (C-G-D-H) FNAF reconstruction: (C) adversarial loss of 0.000229. (G) adversarial loss of 0.00110. (D) adversarial loss of $9.73\cdot10^{-5}$. (H) adversarial loss of 0.000449. }}
  {\includegraphics[width=\linewidth]{MIDL2020_figures/figure1Vic}}
\end{figure}

\subsection{Under-sampling information preservation verification}
To investigate hypothesis 2, we measure the acceptance rate of the adversarial examples based on the information-preservation loss. Shown in \tableref{tab:information_preservation}, a very high acceptance rate is observed across all settings, showing that in most cases the small feature's information is not completely lost through under-sampling, at least for the way we construct the features. We speculate that the same could hold true for real-life abnormalities. 

\figureref{fig:ip_correlation} shows a small negative correlation between IP loss and FNAF loss. In fact, we expect that more information would weaken the attack. However, such negative correlation is weak, indicating that there is no strong association. Therefore the preservation of information alone cannot predict the FNAF-robustness of the model. So the information loss due to under-sampling is a valid but insufficient explanation for the existence of FNAF.

\begin{table}[ht]
 % The first argument is the label.
 % The caption goes in the second argument, and the table contents
 % go in the third argument.
\floatconts
  {tab:information_preservation}%
  {\caption{Information preservation}}%
  {\begin{tabular}{|c|c|c|c|c|}
\hline
                     & Random  & U-Net FNAF & I-RIM FNAF & Robust U-Net FNAF \\ \hline
Acceptance Rate (\%) & 99.82   & 99.72      & 99.76      & 99.34             \\ \hline
IP Loss (MSE)        & 0.00064 & 0.00050    & 0.00051    & 0.00052           \\ \hline
\end{tabular}}
\end{table}

\begin{figure}[ht]
 % Caption and label go in the first argument and the figure contents
 % go in the second argument
\floatconts
  {fig:ip_correlation}
  {\caption{IP loss vs. FNAF loss.}}
  {\includegraphics[width=0.5\linewidth]{MIDL2020_figures/random_ip_correlation.png}}
\end{figure}

\subsection{Location distribution of adversarial features}
We visualize the location distributions of the worst case FNAF on the image in \figureref{fig:location}. There seems to be no apparent pattern to the location of the FNAF. However, the location distributions seems to be similar across non-FNAF-robust models. We investigate this in the next section.

\begin{figure}[ht]
 % Caption and label go in the first argument and the figure contents
 % go in the second argument
\floatconts
  {fig:location}
  {\caption{FNAF location distribution within the 120x120 center crop of the image of (A) U-Net, (B) I-RIM, (C) FNAF-robust U-Net}}
  {\includegraphics[width=0.8\linewidth]{MIDL2020_figures/location_distribution_3models.pdf}}
\end{figure}

\subsection{Transferability of adversarial features across reconstruction networks}
We take FNAF examples from U-Net and apply them to I-RIM, and observe a 89.48\% attack rate. The high transferability is similar to what is observed in \citet{goodfellow} and \citet{alcorn2019strike}. This is indicating that the training data does not capture the distribution of FNAF.

\subsection{Generalization to real-world abnormalities}
A musculoskeletal (MSK) imaging trained M.D. inspects and identifies abnormalities of clinical relevance in 51 volumes from the validation set. The abnormalities include cartilage lesions, meniscal tears, and meniscal degenerations. 

The results in \tableref{tab:real_world} show that the FNAF-robust U-Net is marginally better out of the small number of abnormalities found. Although further extensive evaluation is needed, this is an encouraging result, considering that there is no guarantee that the synthetic feature would look like real-world abnormalities. The detailed comments of the abnormality findings are included in Appendix~\ref{appendix:comments}. An example of the results is shown in \figureref{fig:real_world}.

We suggest the marginal improvements may be explained by the semantic difference between FNAF and real-world abnormalities, although it certainly requires further investigation. Ideally, we want to construct the space of FNAF to be representative of not only the size but also the semantics of real-world abnormalities. We have two ideas on how one could improve FNAF to be more realistic for future works: 1. Relax the pixel constraint more so that the FNAF space can include real-world abnormalities. 2. Model the abnormality features by introducing domain knowledge. Moreover, it is worth noting that FNAF might not even need to be too realistic for deep learning models to generalize. From our experiments, we observe that by training on our imperfect FNAF, one can force convolution filters to be more sensitive to small features. Overall, we think this is the reason for the observed marginal real-world improvements, and it is indicative of a promising direction to move forward to improve clinical robustness.

\section{Conclusions}
The connection between FNAF to real-world abnormalities is analogous to the connection between lp-bounded adversarial perturbations and real-world natural images. In the natural images sampled by non-adversary, lp-bounded perturbations most likely do not exist. But their existence in the pixel space goes beyond security, as they reveal a fundamental difference between deep learning and human vision \cite{bug_feature}. Lp-bounded perturbations violate the human prior: humans see perturbed and original images the same. FNAF violate the reconstruction prior: an algorithm should recover (although it may be impossible) all features. We relax this prior to only small features, which often are the most clinically relevant. Therefore, the failure of deep learning reconstruction models to reconstruct FNAF is important even if FNAF might not be representative of the real-world abnormalities. Lp-bounded perturbations inspired works that generate more realistic attacks, and we hope to bring the same interest in the domain of MRI reconstruction. 
Furthermore, we show the possibility of reconstructing FNAF with adversarial training. In our work, FNAF are constrained to be 10 connected pixels. Arguably, this is smaller than most real-world abnormality features. We believe this indicates that real life abnormalities can be reconstructed from the information preserved through under-sampling.

In this work, we investigate two hypothesis for the false negative problem in deep-learning-based MRI reconstruction. By developing the FNAF adversarial robustness framework, we show that this problem is difficult, but not impossible. Within this framework, there is potential to bring the extensive theoretical and empirical ideas from the adversarial robustness community, especially in the area of provable defenses \cite{wong2018provable, mirman2018differentiable, raghunathan2018certified, balunovic2020adversarialgap} to tackle the problem. We also hope to inspire future work in the direction of defining a better (realistic) search space for the FNAF, towards generalization to real-world abnormalities. We believe that the findings from Appendix~\ref{appendix:comments} can serve as a validation set for future work.

\begin{table}[ht]
 % The first argument is the label.
 % The caption goes in the second argument, and the table contents
 % go in the third argument.
\floatconts
  {tab:real_world}%
  {\caption{Abnormality Reconstructions}}%
  {\begin{tabular}{|c|c|c|}
\hline
                  & Cartilage Lesion Rate & Meniscus Lesion Rate \\ \hline
U-Net             & 1/8                   & 8/9                  \\ \hline
FNAF-robust U-Net & 3/8                   & 9/9                  \\ \hline
\end{tabular}}
\end{table}

\begin{figure}[ht]
 % Caption and label go in the first argument and the figure contents
 % go in the second argument
\floatconts
  {fig:real_world}
  {\caption{(A) Ground truth: small cartilage lesion in femur. (B) U-Net: Area of cartilage lesion not defined and resembles increased signal intensity. (C) FNAF-robust U-Net: Cartilage lesion preserved but less clear.}}
  {\includegraphics[width=\linewidth]{MIDL2020_figures/real_world.png}}
\end{figure}



% Acknowledgments---Will not appear in anonymized version
\midlacknowledgments{We would like to thank
Claudia Iriondo for the help with
the project and fruitful discussions. We would also like to thank Patrick Putzky and his team for releasing the implementation models for IRIM and pointing us to release page.
Ultimately, we would like to thank the reviewers for their constructive feedback and their
efforts towards improving our manuscript.}


%\bibliography{midl-samplebibliography}
\bibliography{cheng20}


\appendix
\newpage

\section{Detailed Comments of Real-World Abnormalities}\label{appendix:comments} 
\begin{table}[ht]
\floatconts
  {tab:MSK comments}%
  {\caption{Comments of the MSK radiologist involved in the study. Cases where FNAF-robust U-Net improves compared to U-Net are bolded. }}%
  {\resizebox{0.81\columnwidth}{!}{\begin{tabular}{|p{1cm}|p{1.4cm}|p{4cm}|p{4cm}|p{4cm}|}
\hline
 \cellcolor[gray]{0.9}File&	 \cellcolor[gray]{0.9} Slice number  &	 \cellcolor[gray]{0.9}Comments on ground truth                   & 	 \cellcolor[gray]{0.9}Comments on U-Net reconstruction	                                          &  \cellcolor[gray]{0.9}Comments on FNAF-robust U-Net reconstruction \\
\hline
7	      &27	  &           Signal change                            & 	Original lesion preserved but less clear                	&\textbf{Original lesion preserved}\\ \hline
26	    &16	  &           Cartilage lesion	                       &  Cartilage lesion in original now looks like signal change	&Cartilage lesion in original now looks like signal change\\\hline
52	    &	    &           Metal artifacts in tibia	               &  No change in metal artifacts	                            &Metal artifacts preserved\\\hline
71	    &23	  &           Cartilage lesion in tibia	               &  Original cartilage lesion not seen	                      &\textbf{Original cartilage lesion preserved but less clear}\\\hline
73	    &23	  &           Intrasubstance degeneration	             &  Intrasubstance degeneration preserved	                    &Intrasubstance degeneration preserved\\\hline
107	    &16	  &           Cartilage lesion in femur	               &  Original cartilage lesion not seen	                      &Original cartilage lesion not seen\\\hline
114	    &26	  &           Vertical tear in meniscus	               &  Original tear preserved but less clear	                  &Original tear preserved but less clear\\\hline
178	    &14-21&           Meniscectomy	                           &  Menisectomy preserved	                                    &Menistectomy preserved\\\hline
196	    &26	  &           Horizontal meniscal tear	               &  Meniscal tear preserved	                                  & Meniscal tear preserved \\\hline
201	    &25	  &           Signal change in femoral cartilage	       &  Cartilage lesion not preserved	                          &\textbf{Cartilage lesion preserved but less clear}\\\hline
267	    &24	  &           Meniscal tear	                           &  Original tear preserved but less clear	                  &Original tear preserved but less clear\\\hline
280	    &22	  &           Cartilage lesion in tibia	               &  Original lesion not preserved	                            &Original lesion not preserved\\\hline
314	    &14-20&           Meniscal degeneration/menisectomy	       &  Meniscal degeneration preserved	                          &Meniscal degeneration preserved\\\hline
325	    &24	  &           Signal change in cartilage	             &  Original cartilage lesion not preserved	                  &\textbf{Signal change in cartilage partially preserved}\\\hline
356	    &21	  &           Cartilage lesion	                       &  Original lesion preserved but not clear	                  &\textbf{Original cartilage lesion preserved}\\\hline
464	    &26	  &           Intrasubstance degeneration	             &  Intrasubstance degeneration preserved but not clear	      &\textbf{Intrasubstance degeneration preserved}\\\hline
480	    &21	  &           Cartilage lesion 	                       &  Cartilage lesion not preserved	                          &Cartilage lesion not preserved\\\hline
528	    &28	  &           Intrasubstance degeneration	             &  Intrasubstance degeneration not preserved	                &\textbf{Intrasubstance degeneration preserved}\\
\hline
\end{tabular}}}
\end{table}
% This is a complete version of a proof sketched in the main text.
\end{document}
