MOSS: Learning from Multiple Organs Via Self-Supervision

Mohammad Reza Hosseinzadeh Taher, Junpyo Hong, Ravi Soni, Gopal Avinash

Abstract: Human anatomy, with its inherent structural consistency across various organs, provides a unique foundation for medical imaging, presenting two key properties: inter-organ discrimination, where each organ displays unique patterns compared to others (e.g., chest vs. hand vs. leg), and intra-organ discrimination, where subtle variations within the same organ across different patients are apparent (e.g., left vs. right hand). We envision developing a medical imaging model that leverages these foundational properties to enhance anatomical “understanding” and establish a “robust” framework for medical imaging. As our first step toward realizing this vision, we introduce MOSS (Multi-Organ Self-Supervised learning), pretrained on 23 diverse organs, that leverages these unique anatomical properties by learning both global anatomical patterns and fine-grained variations to achieve effective inter-organ and intra-organ discrimination. Our experiments across a myriad of tasks in zero-shot, full fine-tuning, and few-shot transfer settings demonstrate that MOSS not only provides a semantically meaningful embedding space that exhibits both inter-organ and intra-organ discrimination but also offers transferable representations that are generalizable to different tasks and robust to low-data regimes, outperforming large-scale fully-supervised and self-supervised medical models. This performance underscores the significance of anatomical understanding through our MOSS, which capitalizes on the inherent structural consistency across multiple organs.