Visual Information Oriented Knowledge Graph

Jinglei Lou, Ang Li, Yu Cao, Mingkai Chen, Mingzi Chen, Xuguang Zhang, Bin Kang

Abstract: Knowledge graphs (KGs) have been widely applied in many IoT application fields. However, existing knowledge graphs primarily focus on simple structured data, ignoring complex data types that are increasingly popular in artificial intelligence domain, such as image, video or text. In this paper, we propose a new pipeline for visual knowledge graph(VKG), in which visual relation detection and knowledge graph construction are integrated into an end-to-end network structure. The proposed network includes three parts 1) object localization, 2) visual relation detection, 3) the construction of visual knowledge graph by relation triples. In particular, we use information processed by object detection to predict the relation between entities on visual relation detection and obtain the knowledge triples. According to the visual knowledge triples, we visualize the constructed visual knowledge graph. The superiority of visual relation detection are shown in the experiments over competitive baselines on the common datasets. Moreover, we also show the performance of visual knowledge graphs based on the result of relation detection.