Beyond Structured Attributes: Image-Based Predictive Trends for Chest X-Ray Classification
Keywords: chest x-ray classification, domain shift, generalization, image statistics
Abstract: A commonly emphasized challenge in medical AI is the drop in performance when testing on data from institutions other than those used for training. However, even if models trained on distinct datasets perform similarly well overall, they may still exhibit other systematic differences. Here, we study these potential dataset-centric prediction variations using two popular chest x-ray datasets, CheXpert (CXP) and MIMIC-CXR (MMC). While CXP-trained models generally perform better on CXP-test splits and vice versa, this performance decrease is not uniform across individual images. Critically, we find that these image-level variations are not random but can be predicted well above chance even for pathologies where the overall performance gap is small, suggesting that there are systematic tendencies of models trained on different datasets. Furthermore, these "predictive tendencies" are not solely explained by image statistics or attributes like radiographic position or patient sex, but rather are pathology-specific and related to higher-order image characteristics. Our findings stress the complexity of AI robustness and generalization, highlighting the need for a nuanced approach that especially considers the diversity of pathology presentation.
Submission Number: 300
Loading