Semi-Supervised Medical Image Segmentation via Knowledge Mining from Large Models
Keywords: Foundation Model, Knowledge Mining, Segment Anything Model, Segmentation, Semi-Supervised Learning
TL;DR: We propose a strategic knowledge mining method as a novel interaction mechanism between large and small models for semi-supervised learning on sparsely labeled medical segmentation datasets.
Abstract: Large-scale vision models like SAM possess extensive visual knowledge, but their application to specialized tasks like medical image segmentation is often hindered by their general nature and the computational challenges associated with training and finetuning. Locally hosted small models such as U-Net++, designed for specific tasks, struggle with limited performance due to sparse labeled datasets. This study introduces a strategic knowledge mining method as a novel interaction mechanism between large and small models. Our method utilizes SAM’s broad visual understanding to enhance the specialized capabilities of locally hosted small deep learning models. Specifically, we trained a U-Net++ model on a limited labeled dataset and extend its capabilities by converting outputs (masks) produced in unlabeled images into prompts, to extract relevant knowledge from SAM. This process not only harnesses SAM’s generalized visual knowledge but also iteratively improves SAM’s prediction to cater specialized medical segmentation tasks via UNet++. The mined knowledge, serving as ‘pseudo labels’, enriches the training dataset, enabling the fine-tuning of the local network. Applied to the Kvasir SEG and COVID-QU-Ex datasets which consist of gastrointestinal polyp and lung Xray images respectively, our proposed method consistently enhanced the segmentation performance on Dice by 3% and 1% respectively over the baseline U-Net++ model, when the same amount of labelled data were used during training (75% and 50% of labelled data). Remarkably, our proposed method surpassed the baseline U-Net++ model even when the latter was trained exclusively on labeled data (100% of labelled data). These results underscore the potential of knowledge mining to overcome data limitations in specialized models by leveraging the broad, albeit general, knowledge of large-scale models like SAM, all while maintaining operational efficiency essential for clinical applications.The code of our method is publicly available at https://anonymous.4open.science/r/Knowledge-Mining-from-Large-Models-C7FE.
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 10624
Loading