Visual entity linking

Neha Tilak, Sunil Gandhi, Tim Oates

Abstract: Entity linking is the task of identifying entities like people and places in textual data and linking them to corresponding entities in a knowledge base. In this paper we solve a visual equivalent of this task called visual entity linking. The goal is to link regions of images to corresponding entities in knowledge bases. Visual entity linking will enable computers to better understand visual content and thus can be used in tasks like image retrieval and visual question answering. More specifically, we propose a novel approach for linking image regions to entities in Dbpedia and Freebase. First, we select candidate entities using an automatic image description generation algorithm. We then extract image regions using object detection methods and compare them to depictions of entities in a knowledge base. We evaluate our approach on the Flickr8k dataset through surveys on Amazon Mechanical Turk, and present an extensive analysis to identify the sources of errors in our system.