Attention Enhanced Network with Semantic Inspector for Medical Image Report Generation

Yihan Lin, Qian Tang, Hao Wang, Cheng Huang, Favour Ekong, Xiangxiang Wang, Xiao Feng, Yongbin Yu

Abstract: Medical report generation can be helpful in diagnoses. Despite the previous efforts of researchers, current models still need improvements in the extraction of image features and quality of generated reports. In this paper, we propose an attention enhanced network with semantic inspector (AENSI) as a new automatic medical report generation model, which serves to help doctors get a high-quality report. For the model, we propose double-weighted multi-head attention as our attention module, where different heads are aggregated with double weights (DWMHA) to enhance its power in catching subtle features and drawing correlations between images and texts. To prevent the drawback of imprecise multi-label classification modules used in current generation models, we design a novel module following decoder that treats tags as inspectors of the generated reports, namely Tag Inspector, as a substitute for the previous classification module. Experimental results of AENSI achieve to the level of state-of-the-art. On IU X-ray, our model surpasses all previous works on every metrics; on PEIR Gross, our model ranks first on BLEU-4 and ROUGE and closely approaches the best on other metrics.