Editing with AI: How Doctors Refine LLM-Generated Answers to Patient Queries

Rahul Sharma, Pragnya Ramjee, Kaushik Murali, Mohit Jain

Keywords: Human-AI Collaboration, Clinical Communication, Direct vs. Indirect Editing, Mixed-Methods Study, Ophthalmology, Cataract Surgery

TL;DR: We conducted a mixed-methods study with cataract surgeons to compare three paradigms of human-AI coauthoring for clinical communication: writing from scratch, direct editing LLM drafts, and instruction-based editing.

Abstract: Patients frequently seek information during their medical journeys, but the rising volume of digital patient messages has strained healthcare systems. Large language models (LLMs) offer promise in generating draft responses for clinicians, yet how physicians refine these drafts remains underexplored. We present a mixed-methods study with nine ophthalmologists answering 144 cataract surgery questions across three conditions: writing from scratch, directly editing LLM drafts, and instruction-based indirect editing. Our quantitative and qualitative analyses reveal that while LLM outputs were generally accurate, occasional errors and automation bias revealed the need for human oversight. Contextualization---adapting generic answers to local practices and patient expectations---emerged as a dominant form of editing. Editing workflows revealed trade-offs: indirect editing reduced effort but introduced errors, while direct editing ensured precision but with higher workload. We conclude with design and policy implications for building safe, scalable LLM-assisted clinical communication systems.

Submission Number: 109