Automated multimodal sensemaking: Ontology-based integration of linguistic frames and visual data

Fiorela Ciroku, Stefano De Giorgis, Aldo Gangemi, Delfina Sol Martinez Pandiani, Valentina Presutti

Abstract: Highlights•Enables complex querying of Visual Genome dataset for visual content knowledge.•Connects frame evocation to image regions for deep exploration.•Formalizes Visual Sense Ontology (VSO) for multimodal data enhancement.•Extracts and connects linguistic frames, values, and emotions to images.•Advances frame evocation and multimodal sensemaking with transparency.