\section{Conclusions}

In this paper, we propose a Memory U-Net that takes the advantage of the U-Net and memory networks for lesion instance segmentation.
This is the first time that a unified model is developed for simultaneous lesion counting and individual lesion segmentation.
We use the strong representation learning ability of U-Net to extract voxel-wise feature vectors.
We also develop a novel key-value structured memory network with tailored memory updating mechanism and variance reducing loss function, which can alleviate the long-tailed issue in offset estimation.
This is also the first time that memory network is used as an alternative code book for generalized Hough voting.
We believe that the proposed method can not only provide a good baseline for MS lesion instance segmentation, but also can promote related research on other medical image applications.
However, the major drawback of the method is that k-means is used to cluster the individual lesion volume, which is sub-optimal, as lesions vary greatly in terms of the shape and size.
Thus, in future work, we will consider embedding the voxel association process into an end-to-end learning framework.