Human-centric annotation of multi-modal data: A framework perspective

BELIVEAU, Yann-Romain KECHABIA, Cyril RAY, Maeve PETIT, Fabienne LAJONCHERE, PUENTES John

Keywords: Multi-modal data, Annotation, Data-Centric Artificial In- telligence, Human-Centric design, Human-In-The-Loop, Decision support

TL;DR: This paper shows data annotation as a human-centric process —through a naval defense use case—and demonstrates how it can support decision-making, reduce cognitive overload, and maintain situational awareness in high-stakes environments.

Abstract: Data annotation is a cognitively rich process shaped by perception, judgment, and variability. However, it is traditionally viewed as a mechanical and time-consuming task for humans, and is therefore often automated or outsourced. This paper considers annotation as a Human-Centric cognitive process and proposes a generic framework in which annotation supports humans in interpreting complex multimodal data and operational information. Through a taxonomy of human-in-the-loop semi-automatic pipelines, we show how AI-generated annotations can interact with humans. We apply this framework to a naval defense use case, specifically maritime surveillance activity. In this high-stakes environment, where operators must process continuous and heterogeneous information streams, annotation acts as a cognitive aid. It can ultimately become a decision-support tool that reduces cognitive load and preserves situational awareness, rather than remaining a simple data-labeling mechanism.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 9