SegMaST: Mamba-based Spatio-Temporal Modeling to Improve Longitudinal Detection and Segmentation

Aswathi Varma, Jonas Weidner, Laurin Lux, Cosmin I. Bercea, Mark Mühlau, Jan Kirschke, Benedikt Wiestler, Daniel Rueckert

Keywords: Longitudinal Segmentation, Spatio-Temporal Attention, Mamba, Imbalance Aware Learning

TL;DR: SegMaST is a Mamba-based spatio-temporal model that jointly segments baseline and new lesions while handling real-world progression imbalance.

Abstract: Longitudinal medical image segmentation is fundamental for quantifying disease progression and evaluating treatment efficacy. However, two critical challenges persist: First, methods that jointly segment baseline and follow-up images remain underexplored, often missing the contextual benefits of simultaneous assessment and lacking longitudinal consistency. Second, real-world datasets typically exhibit severe class imbalance between stable and progressive scans — an issue frequently neglected by existing models. To address these limitations, we propose SegMaST, a novel Mamba-based spatio-temporal framework. Unlike conventional approaches that treat timepoints in isolation, SegMaST leverages cross-temporal information and spatial correspondences to jointly segment the initial baseline mask and explicitly localize new pathologies in follow-up scans. Additionally, we introduce an imbalance-aware loss accumulation strategy to enhance robustness in realistic clinical settings. On longitudinal Multiple Sclerosis and Glioma cohorts, SegMaST outperforms established CNN- and attention-based baselines for follow-up segmentation (mean follow-up Dice MS in-house $0.536$, MSSEG-2 $0.620$, and Glioma $0.631$) and lesion detection (F1 in-house $0.688$, MSSEG-2 $0.723$), while maintaining state-of-the-art accuracy in baseline segmentation (Dice: $0.617$ MS, $0.844$ Glioma).

Primary Subject Area: Segmentation

Secondary Subject Area: Application: Neuroimaging

Registration Requirement: Yes

Visa & Travel: Yes

Read CFP & Author Instructions: Yes

Originality Policy: Yes

Single-blind & Not Under Review Elsewhere: Yes

LLM Policy: Yes

Midl Latex Submission Checklist: Ensure no LaTeX errors during compilation., Replace NNN with your OpenReview submission ID., Includes \documentclass{midl}, \jmlryear{2026}, \jmlrworkshop, \jmlrvolume, \editors, and correct \bibliography command., Did not override options of the hyperref package., Did not use the times package., Use the correct spelling and format, avoid Unicode characters, and use LaTeX equivalents instead., Any math in the title and abstract must be enclosed within $...$., Did not override the bibliography style defined in midl.cls and did not use \begin{thebibliography} directly to insert references., Avoid using \scalebox; use \resizebox when needed., Included all necessary figures and removed *unused* files in the zip archive., Removed special formatting, visual annotations, and highlights used during rebuttal., All special characters in the paper and .bib file use LaTeX commands (e.g., \'e for é)., No separate supplementary PDF uploads., Acknowledgements, references, and appendix must start after the main content.

Submission Number: 117