Mask What Matters: Controllable Text-Guided Masking for Self-Supervised Medical Image Analysis

ICLR 2026 Conference Submission25040 Authors

Keywords: Medical Image Analysis, Self-Supervised Learning, Vision-Language Models

Abstract:

The scarcity of annotated data in specialized domains such as medical imaging presents significant challenges to training robust vision models. While self-supervised masked image modeling (MIM) offers a promising solution, existing approaches largely rely on random high-ratio masking, leading to inefficiency and poor semantic alignment. Moreover, region-aware variants typically depend on reconstruction heuristics or supervised signals, limiting their adaptability across tasks and modalities.

We propose Mask What Matters, a controllable text-guided masking framework for self-supervised medical image analysis. By leveraging vision-language models for prompt-based region localization, our method flexibly applies differentiated masking to emphasize diagnostically relevant regions while reducing redundancy in background areas. This controllable design enables better semantic alignment, improved representation learning, and stronger cross-task generalizability.

Comprehensive evaluation across multiple medical imaging modalities, including brain MRI, chest CT, and lung X-ray, shows that Mask What Matters consistently outperforms existing MIM methods (e.g., SparK), achieving gains of up to +3.1 percentage points in classification accuracy, +1.3 in box average precision (BoxAP), and +1.1 in mask average precision (MaskAP) for detection. Notably, it achieves these improvements with substantially lower overall masking ratios (e.g., 40% vs. 70%), highlighting its efficiency and flexibility.

This work demonstrates that controllable, text-driven masking can enable semantically aligned and generalizable self-supervised learning, advancing the development of robust vision models for medical image analysis.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 25040