Improving Survival Prediction of Head-and-Neck Cancer with Medical Image, Foundation Models and Multi-modal Fusion

Haotian Zhang, Yiming Liu, Yile Sun, Liyue Shen, Lise Wei

Keywords: HNSCC, medical foundation models, PET/CT, multimodal fusion, survival prediction

TL;DR: We boost HNSCC survival prediction by inserting frozen CT/PET foundation-model embeddings (projected with small linear heads) into a DeepMTS+DAFT baseline and late-fusing them, which raises C-index and reduces variance under 5-fold CV.

Track: Findings

Abstract:

Accurate survival prediction in head and neck squamous cell carcinoma (HNSCC) is critical for guiding treatment stratification but remains difficult due to small cohort sizes and the suboptimal integration of multimodal medical data.
In particular, imaging features are high-dimensional, while clinical covariates are low-dimensional, making effective multimodal fusion non-trivial.

To address these limitations, we propose to leverage pretrained modality-specific medical image foundation models (FMs), including those for CT and PET, to improve image representation learning under small-sample constraints. These models can augment a multimodal fusion baseline to enhance the structured fusion of imaging and clinical features.
Specifically, FM-derived embeddings are passed through compact linear projection heads and fused by direct concatenation immediately before the survival prediction head.

We systematically study the projection dimensionality and modality composition (CT-only, PET-only, CT+PET) and evaluate the performance using the average concordance index (C-index) and time-dependent AUROC under five-fold cross-validation.
Preliminary results demonstrate that image embeddings extracted from pretrained medical foundation models consistently improve C-index and time-dependent AUROC, and reduce their fold-to-fold variance, with the CT+PET setting providing the most robust gains.
These findings suggest that FM-based multimodal survival models can enhance risk stratification and ultimately support personalized treatment adaptation in HNSCC.

General Area: Applications and Practice

Specific Subject Areas: Foundation Models, Medical Imaging

Data And Code Availability: No

Ethics Board Approval: No

Entered Conflicts: I confirm the above

Anonymity: I confirm the above

Submission Number: 124