Vision-knowledge fusion model for multi-domain medical report generation

Dexuan Xu, Huashi Zhu, Yu Huang, Zhi Jin, Weiping Ding, Hang Li, Menglong Ran

Abstract: Highlights•An automatic method to construct domain knowledge graph based on medical standards.•New knowledge-based attention for image and graph.•A vision-knowledge fusion model with triples restoration.•A more reasonable knowledge-based metrics for report generation.•Well performance on two datasets from different domains.