Contrastive VQ Priors for Multi-Class Plaque Segmentation via SAM Adaptation

TMLR Paper7140 Authors

Abstract:

Accurate plaque subtype segmentation in coronary CT angiography (CCTA) is clinically relevant yet remains difficult in practice, where annotations are scarce and the visual evidence for non-calcified lesions is subtle and highly variable. Meanwhile, segmentation foundation models such as SAM provide strong robustness from large-scale pretraining, but their benefits do not reliably transfer to private CCTA tasks under naïve fine-tuning, especially for multi-class plaque taxonomy.
We present a targeted strategy to transfer SAM's segmentation robustness to a private CCTA setting by injecting a task-specific, texture-aware prior into the SAM feature stream.
Our framework is two-stage: (i) we learn a discrete latent prior from the private CCTA data using a vector-quantized autoencoder, and structure it with supervised contrastive learning to emphasize hard class boundaries; (ii) we fuse this prior into a SAM-based encoder through a query-based feature-aware cross-attention module, and decode with a multi-class head/decoder tailored for plaque taxonomy.
On the private CCTA benchmark, our approach consistently improves plaque subtype delineation and outperforms strong medical baselines (nnU-Net, TransUNet) as well as SAM-family adaptations (including Medical SAM variants and CAT-SAM). Ablations verify the roles of (a) contrastively-structured discrete priors, (b) attention-based retrieval versus additive fusion, and (c) multi-class decoding for SAM-style models.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Chenyu_You1

Submission Number: 7140