
\begin{abstract}
Confluent lesions usually occur when pathologically distinct lesions grow close to each other and form a large spatially-connected lesion.   
These confluent lesions pose a great challenge for subsequent image analysis and disease diagnosis, as individual lesions are difficult to separate and segment.
In this paper, we propose a Memory U-Net that takes advantage of recent fully convolutional neural network U-Net and memory networks, to resolve the issue.
The main idea is that we develop a hybrid model with a U-Net for feature extraction and a memory network as the alternative code book for generalized Hough voting.
To alleviate the GPU memory overhead brought by the large code book, we decompose the large code book into three smaller ones, where each one of them accounts for voting in one specific direction.  
Through voxel-wise voting, a density map of lesion locations can be obtained by aggregating votes from all lesion voxels, and this density map is further used to generate final instance segmentation results.
Experiments on a large-scale cross-sectional multiple sclerosis study verifies the efficiency and the effectiveness of the proposed method.

\end{abstract}

\begin{keywords}
Memory Network, Hough Voting, Instance Segmentation, Multiple Sclerosis Lesion
\end{keywords} 