VR-DiagNet: Medical Volumetric and Radiomic Diagnosis Networks with Interpretable Clinician-like Optimizing Visual Inspection

Shouyu Chen, Tangwei Ye, Lai Zhong Yuan, Qi Zhang, KE LIU, Usman Naseem, Ke Sun, Nengjun Zhu, Liang Hu

Abstract: Interpretable and robust medical diagnoses are essential traits for practicing clinicians. Most computer-augmented diagnostic systems suffer from three major problems: non-interpretability, limited modality analysis, and narrow focus. Existing frameworks can either deal with multimodality to some extent but suffer from non-interpretability or partially interpretable but provide a limited modality and multifaceted capabilities. Our work aims to integrate all these aspects in one complete framework to fully utilize the full spectrum of information offered by multiple modalities and facets. We propose our solution via our novel architecture VR-DiagNet, consisting of a planner and a classifier, optimized iteratively and cohesively. VR-DiagNet simulates the perceptual process of clinicians via the use of volumetric imaging information integrated with radiomic features modality; at the same time, it recreates human thought processes via a customized Monte Carlo Tree Search (MCTS) which constructs a volume-tailored experience tree to identify slices of interest (SoIs) in our multi-slice perception space. We conducted extensive experiments across two diagnostic tasks comprising six public medical volumetric benchmark datasets. Our findings showcase superior performance, as evidenced by heightened accuracy and area under the curve (AUC) metrics, reduced computational overhead, and expedited convergence while conclusively illustrating the immense value of integrating volumetric and radiomic modalities for our current problem setup.

Primary Subject Area: [Experience] Multimedia Applications

Secondary Subject Area: [Content] Multimodal Fusion

Relevance To Conference: Grounded in interdisciplinary research at the intersection of engineering and medical disciplines, this paper introduces a novel approach to three-dimensional medical image multimodal diagnosis. This method harnesses voxel and radiomics features concurrently, emulating the diagnostic process of clinicians' visual inspection, to deliver accurate and interpretable diagnoses.

Submission Number: 660