Observer: Creation of a Novel Multimodal Dataset for Outpatient Care Research

Download PDF
Open Webpage

Kevin B. Johnson, Basam Alasaly, Kuk Jin Jang, Eric Eaton, Sriharsha Mopidevi, Ross Koppel

Published: 19 May 2025, Last Modified: 06 Jan 2026CrossrefEveryoneRevisionsCC BY-SA 4.0
Abstract: h3>Abstract</h3> <h3>Objective</h3> <p>To support ambulatory care innovation, we created <i>Observer</i>, a multimodal dataset comprising videotaped outpatient visits, electronic health record (EHR) data and structured surveys. This paper describes the data collection procedures and summarizes the clinical and contextual features of the dataset.</p><h3>Materials and Methods</h3> <p>A multistakeholder steering group shaped recruitment strategies, survey design, and privacy-preserving design. Consented patients and primary care providers (PCPs) were recorded using room-view and egocentric cameras. EHR data, metadata and audit logs were also captured. A custom de-identification pipeline, combining transcript redaction, voice masking, and facial blurring, ensured video and EHR HIPAA compliance.</p><h3>Results</h3> <p>We report on the first 100 visits in this continually growing dataset. Thirteen PCPs from four clinics participated. Recording the first 100 visits required approaching 210 patients, from which 129 consented (61%), with 29 patients missing their scheduled encounter after consenting. Visit lengths ranged from 5 to 100 minutes, covering preventive care to chronic disease management. Survey responses revealed high satisfaction: 4.24/5 (patients) and 3.94/5 (PCPs). Visit experience was unaffected by the presence of video recording technology.</p><h3>Discussion</h3> <p>We demonstrate the feasibility of capturing rich, real-world primary care interactions using scalable, privacy-sensitive methods. Room layout and camera placement were key influences on recorded communication and are now added to the dataset. The <i>Observer</i> dataset enables future clinical AI research/development, communication studies, and informatics education among public and private user groups.</p><h3>Conclusion</h3> <p>Observer is a new, shareable, real-world clinic encounter research and teaching resource with a representative sample of adult primary care data.</p>