Parameter Efficient Fine-Tuning of Segment Anything Model
Keywords: segment anything, peft, biomedical image segmentation
TL;DR: This paper highlights the application of PEFT methods to efficiently fine-tune vision foundation models like SAM, MicroSAM and MedicoSAM for better biomedical image segmentation, reducing computational costs and improving performance on diverse data.
Abstract: Segmentation is an important analysis task for biomedical images, enabling the study of individual organelles, cells or organs. Deep learning has massively improved segmentation methods, but challenges remain in generalization to new conditions, requiring costly data annotation.
Vision foundation models, such as Segment Anything Model (SAM), address this issue through broad segmentation capabilities. However, these models still require finetuning on annotated data, although with less annotations, to achieve optimal results for new conditions. As a downside, they require more computational resources. This makes parameter-efficient finetuning (PEFT) relevant for their application. We contribute the first comprehensive study of PEFT for SAM applied to biomedical segmentation by evaluating 9 PEFT methods on diverse datasets. We also provide an implementation of QLoRA for vision transformers and a new approach for resource-efficient finetuning of SAM.
Primary Subject Area: Foundation Models
Secondary Subject Area: Segmentation
Paper Type: Methodological Development
Registration Requirement: Yes
Reproducibility: https://github.com/computational-cell-analytics/peft-sam
Visa & Travel: Yes
Submission Number: 184
Loading