HectoMixNet: Advancing Automated Head and Neck Tumor Segmentation with Multicenter PET/CT Data

Moona Mazher, Steven Niederer, Abdul Qayyum

Keywords: Bidirectional Quasiseparable Mixing, State-Space Models, Head and Neck Cancer Tumor Segmentation, Lymph Node Detection, PET/CT Imaging, Radiotherapy Planning, HECKTOR Challenge

Abstract: Head and Neck (H&N) cancers represent one of the most prevalent malignancies worldwide, with accurate delineation of primary tumors and metastatic lymph nodes being essential for radiotherapy planning and treatment outcome prediction. Manual segmentation, though standard, is time-consuming and prone to inter-observer variability, motivating the development of automated deep learning–based methods. In this study, we propose HectoMixNet, a novel 3D encoder–decoder framework augmented with a bidirectional quasiseparable mixing module for robust segmentation of primary tumors and lymph nodes in multimodal PET/CT imaging. The proposed module efficiently integrates local anatomical detail with global volumetric context, addressing the challenges of heterogeneous and spatially complex lesions. We evaluate HectoMixNet on the HECKTOR 2025 Task 1 dataset, the largest multicentric benchmark for H&N cancer analysis to date, comprising over 1,200 patients from 11 international centers. Our model achieved strong performance, with a GTVp Dice score of 0.8812, GTVn Dice of 0.8246, and GTVn F1 of 0.7273 on the validation and leaderboard sets. Compared to state-of-the-art baselines such as 3D Mamba and xLSTM, HectoMixNet demonstrated superior lymph node segmentation accuracy, highlighting the importance of bidirectional quasiseparable mixing for modeling long-range spatial dependencies. These results establish HectoMixNet as an effective and clinically relevant tool for automated H&N cancer segmentation in large-scale, heterogeneous imaging cohorts.

Submission Number: 11