Emulating Oncologists' Gaze for Predicting Treatment Response through Multimodal Imaging

NeurIPS 2023 Workshop Gaze Meets ML Submission19 Authors

Keywords: Predicting Treatment Response, Density-based isometric mapping, Multimodal imaging

TL;DR: The study emulate oncologists' prognostic decisions by looking at lung tumors in various imaging modalities and using density-based isometric mapping.

Abstract: We used three datasets to imitate oncologists' decisions about the prognostic response by looking at lung tumors in different imaging modalities, i.e., PET and CT. We extract comprehensive visual features, radiomics, from the tumors' region and then reduce their size with a density-based isometric mapping while preserving their main visual characteristics. We apply the Parzen-Rosenblatt (PR) constrain to modify isometric mapping. For the comparison, we use two metrics, binary classification and Cox proportional hazard models, to avoid biases in the comparison. We achieved prediction accuracy comparable to newly and commonly established methods. We successfully predict patient outcomes in response to therapy and imitate the oncologist's attention over multimodal images.

Submission Type: Extended Abstract

Submission Number: 19