Deep Reinforcement Exemplar Learning for Annotation Refinement

Yuexiang Li, Nanjun He, Sixiang Peng, Kai Ma, Yefeng Zheng

Abstract: Due to the inter-observer variation, the ground truth of lesion areas in pathological images is generated by majority-voting of annotations provided by different pathologists. Such a process is extremely laborious, since each pathologist needs to spend hours or even days for pixel-wise annotations. In this paper, we propose a reinforcement learning framework to automatically refine the set of annotations provided by a single pathologist based on several exemplars of ground truth. Particularly, we treat each pixel as an agent with a shared pixel-level action space. The multi-agent model observes several paired single-pathologist annotations and ground truth, and tries to customize the strategy to narrow down the gap between them with episodes of exploring. Furthermore, we integrate a discriminator to the multi-agent framework to evaluate the quality of annotation refinement. A quality reward is yielded by the discriminator to update the policy of agents. Experimental results on the publicly available Gleason 2019 dataset demonstrate the effectiveness of our reinforcement learning framework—the segmentation network trained with our refined single-pathologist annotations achieves a comparable accuracy to the one using majority-voting-based ground truth.