Learning from What the Model Forgets: Prototype-Guided Patch-wise Replay for Medical Image Segmentation

ICLR 2026 Conference Submission1206 Authors

Keywords: medical image segmentation, moderately forgettable samples, prototype learning

TL;DR: We improve medical image segmentation by identifying and reinforcing "moderately forgettable" samples near decision boundaries using CLIP-guided prototypes and an adaptive sample bank, reducing false negatives and improving boundary accuracy.

Abstract: Medical image segmentation remains a challenging problem due to the presence of hard positive samples that deviate from class centers and are frequently forgotten during training. These moderately forgettable samples often reside near decision boundaries and exhibit inconsistent learning behavior, contributing to elevated false negative rates and suboptimal boundary delineation. Existing methods lack effective mechanisms to identify and reinforce such samples, especially under patch-wise training constraints imposed by large-volume medical data. We propose an end-to-end online learning framework that systematically mines these moderately forgettable samples. Our method comprises three complementary modules: (1) Text-Guided Fusion, which incorporates CLIP-based text embeddings to guide semantic prototype learning and enhance feature representation; (2) Prototype-Based Scoring, which evaluates sample difficulty across intra-class consistency, inter-class distinction, prediction deviation, and model confidence; and (3) an Online Forgettable Sample Bank, which adaptively retains and replays informative samples through curriculum learning. Experiments on multiple public datasets demonstrate that our approach consistently reduces false negative rates and improves boundary accuracy in clinically challenging scenarios.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 1206