VisionKG: Unleashing the Power of Visual Datasets via Knowledge Graph

Jicheng Yuan, Anh Le-Tuan, Manh Nguyen-Duc, Trung-Kien Tran, Manfred Hauswirth, Danh Le Phuoc

Abstract: The availability of vast amounts of visual data with diverse and fruitful features is a key factor for developing, verifying, and benchmarking advanced computer vision (CV) algorithms and architectures. Most visual datasets are created and curated for specific tasks or with limited data distribution for very specific fields of interest, and there is no unified approach to manage and access them across diverse sources, tasks, and taxonomies. This not only creates unnecessary overheads when building robust visual recognition systems, but also introduces biases into learning systems and limits the capabilities of data-centric AI. To address these problems, we propose the Vision Knowledge Graph (VisionKG), a novel resource that interlinks, organizes and manages visual datasets via knowledge graphs and Semantic Web technologies. It can serve as a unified framework facilitating simple access and querying of state-of-the-art visual datasets, regardless of their heterogeneous formats and taxonomies. One of the key differences between our approach and existing methods is that VisionKG is not only based on metadata but also utilizes a unified data schema and external knowledge bases to integrate, interlink, and align visual datasets. It enhances the enrichment of the semantic descriptions and interpretation at both image and instance levels and offers data retrieval and exploratory services via SPARQL and natural language empowered by Large Language Models (LLMs). VisionKG currently contains 617 million RDF triples that describe approximately 61 million entities, which can be accessed at https://vision.semkg.org and through APIs. With the integration of 37 datasets and four popular computer vision tasks, we demonstrate its usefulness across various scenarios when working with computer vision pipelines.