TRAUMA THOMPSON: A Novel Dataset and Benchmarks For AI Copilots For Humanitarian Operational Medicine

ICLR 2026 Conference Submission20683 Authors

Keywords: Egocentric dataset, action recognition, action anticipation, visual question answering, realism detection, medical AI

Abstract: This paper introduces the Trauma THOMPSON dataset, a novel dataset and benchmarks to foster research towards designing artificial intelligence-based decision-making algorithms specifically suited for life-saving interventions performed by less experienced caregivers. This paradigm is particularly relevant to support humanitarian operational medicine, where the essential resources are either unavailable or significantly restricted. Our dataset includes a total of 3717 high-resolution clips and ground-truth action annotations by medical professionals. The events are unscripted, and the clips are all assigned a specific medical skill relevant to life-saving interventions. There are two skill types: regular procedures with standard medical tools and improvised procedures with daily objects. We have augmented this dataset with additional annotations, including medical visual question answering, hand tracking and object detection. Moreover, we propose a framework for replacing manikins in the dataset with real patients and a realism detection method. Benchmarks are provided for action recognition, action anticipation, and visual question answering (VQA) using a variety of vision models and vision language models (VLMs). We found that MViT v2 is the best performer for action recognition and action anticipation and BLIP for VQA. By consolidating diverse annotations into a single dataset and a framework to create realistic patient images, Trauma THOMPSON dataset offers a foundation for training unified VLMs as AI medics that can perform holistic reasoning and decision-making in disconnected and high-stakes settings to support less experienced first responders. The dataset and codes are available at https://dataverse.harvard.edu/previewurl.xhtml?token=bd66015d-64bc-4203-ad09-5ab5c90832ef.

Primary Area: datasets and benchmarks

Submission Number: 20683