Pathological Visual Question Answering

Xuehai He, Zhuo Cai, Wenlan Wei, Yichen Zhang, Luntian Mou, Eric Xing, Pengtao Xie

Keywords: Pathology Visual Question Answering, Healthcare, Learning to Ignore, Self-supervised Learning

Abstract: Is it possible to develop an "AI Pathologist" to pass the board-certified examination of the American Board of Pathology (ABP)? To build such a system, three challenges need to be addressed. First, we need to create a visual question answering (VQA) dataset where the AI agent is presented with a pathology image together with a question and is asked to give the correct answer. Due to privacy concerns, pathology images are usually not publicly available. Besides, only well-trained pathologists can understand pathology images, but they barely have time to help create datasets for AI research. The second challenge is: due to the fact that it is difficult to hire highly experienced pathologists to create pathology visual questions and answers, the resulting pathology VQA dataset may contain errors such as some questions may not be relevant to the image or the answers are not given correctly. Training pathology VQA models using these noisy or even erroneous data will lead to problematic models that cannot generalize well on unseen images. The third challenge is: the medical concepts and knowledge covered in pathology question-answer (QA) pairs are very diverse while the number of QA pairs available for modeling training is limited. How to learn effective representations of diverse medical concepts based on limited data is technically demanding. In this paper, we aim to address these three challenges. To our best knowledge, our work represents the first one addressing the pathology VQA problem. To deal with the issue that a publicly available pathology VQA dataset is lacking, we create PathVQA, a VQA dataset with 32,795 questions asked from 4,998 pathology images. The questions in PathVQA are similar to those in the ABP tests. To our best knowledge, this is the first dataset for pathology VQA. To address the second challenge, we propose a learning-by-ignoring approach which automatically identifies training examples that have bad-quality and remove them from the training dataset. To address the third challenge, we propose to use crossmodal self-supervised learning to learn powerful visual and textual representations jointly. We perform experiments on our created PathVQA dataset and the results demonstrate the effectiveness of our proposed learning-by-ignoring method and cross-modal self-supervised learning methods.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We develop datasets and methods to perform visual question answering on pathology images.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=aOU5bNR2V