Medical knowledge-based network for Patient-oriented Visual Question Answering

Jian Huang, Yihao Chen, Yong Li, Zhenguo Yang, Xuehao Gong, Fu Lee Wang, Xiaohong Xu, Wenyin Liu

Abstract: Highlights•We introduce a new Patient-oriented medical VQA dataset (P-VQA).•P-VQA covers various aspects of the medical treatment process for patients.•A Medical Knowledge-based VQA Network (MKBN) is proposed for medical VQA.•A medical graph encoder is designed to encode the medical knowledge graph.•Tri-fusion stages are designed to fuse diseases, relations and images in three ways.•Experiments show that MKBN achieves the state-of-the-art performance.