Go to ICLR 2026 Conference homepageMedITok: A Unified Tokenizer for Medical Image Synthesis and Interpretation
Keywords: visual tokenizer, foundation model, general medical
TL;DR: A novel training framework to produce MedITok, the first unified visual tokenizer for medical images, effectively encoding visual details and clinical semantics, achieving SOTA performance across diverse medical modalities and tasks.
Abstract: Advanced autoregressive models have reshaped multimodal AI. However, their transformative potential in medical imaging remains largely untapped due to the absence of a unified visual tokenizer---one capable of capturing fine‑grained visual structures for faithful image reconstruction and realistic image synthesis, as well as rich semantics for accurate diagnosis and image interpretation.
To this end, we present MedITok, the first unified tokenizer tailored for medical images, encoding both low-level structural details and high-level clinical semantics within a unified latent space.
To balance these competing objectives, we introduce a novel two‑stage training framework: a visual representation alignment stage that cold-starts the tokenizer reconstruction learning with a visual semantic constraint, followed by a textual semantic representation alignment stage that infuses detailed clinical semantics into the latent space.
Trained on the meticulously collected large-scale dataset with over 30 million medical images and 2 million image-caption pairs, MedITok achieves state-of-the-art performance on more than 30 datasets across 9 imaging modalities and 4 different tasks. By providing a unified token space for autoregressive modeling, MedITok supports a wide range of tasks in clinical diagnostics and generative healthcare applications.
Model and code are available in the supplementary material.
Supplementary Material: zip
Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)
Submission Number: 8989
Loading