Learning Attention from Multi-Modal Imaging and Text: Application for Lesion Localisation in DWI

Aydan Gasimova, Liang Chen, Paul Bentley, Giovanni Montana, Daniel Rueckert

Abstract: Strokes are one of the leading causes of death and disability in the UK. There are two main types of stroke: ischemic and hemorrhagic, with the majority of stroke patients suffering from the former. During an ischemic stroke, parts of the brain lose blood supply, and if not treated immediately, can lead to irreversible tissue damage and even death. Ischemic lesions can be detected by diffusion weighted magnetic resonance imaging (DWI), but localising and quantifying these lesions can be a time consuming task for clinicians. Work has already been done in training neural networks to segment these lesions, but these frameworks require a large amount of manually segmented 3D images, which are very time consuming to create. We instead propose to use past examinations of stroke patients which consist of DWIs, corresponding radiological reports and diagnoses in order to develop a learning framework capable of localising lesions. This is motivated by the fact that the reports summarise the presence, type and location of the ischemic lesion for each patient, and thereby provide more context than a single diagnostic label. Localisation of lesions is aided by an attention mechanism which implicitly learns which regions within the DWI are most relevant to the classification.

Keywords: brain dwi, lesions, multimodal attention

Author Affiliation: Imperial College London, Warwick University