Preventing unrestricted and unmonitored AI experimentation in healthcare through transparency and accountability

Donnella S. Comeau, Danielle S. Bitterman, Leo Anthony Celi

Abstract: The integration of large language models (LLMs) into electronic health records offers potential benefits but raises significant ethical, legal, and operational concerns, including unconsented data use, lack of governance, and AI-related malpractice accountability. Sycophancy, feedback loop bias, and data reuse risk amplifying errors without proper oversight. To safeguard patients, especially the vulnerable, clinicians must advocate for patient-centered education, ethical practices, and robust oversight to prevent harm.