Advancing Chest X-ray Diagnostics via Multi-Modal Neural Networks with Attention

Douglas Townsell, Tanvi Banerjee, Lingwei Chen, Michael L. Raymer

Abstract: The healthcare field is undergoing a profound shift, with deep learning in AI increasingly augmenting medical expertise in complex and challenging tasks. Our research addresses the challenging task of chest X-ray image diagnostics, a field characterized by multifaceted diagnostic labels and class im-balances in respiratory disease cases. Our approach synergizes a pre-trained image classification neural network with patient and image metadata integration, significantly boosting diagnostic precision. A key aspect of our methodology is the identification of an effective decision boundary to enhance accuracy and reduce false positives. The effectiveness of our model is demonstrated by an average AUC score of 0.84, surpassing existing models and signifying a notable leap in AI’s role in medical diagnostics. This tool stands to aid clinical decision-making, particularly in navigating the complexities of comorbidities in respiratory health.