MedAID-ML: A Multilingual Dataset of Biomedical Texts for Detecting AI-Generated Content

Patrick Styll, Leonardo Campillos-Llanos, Jorge Fernández-García, Isabel Segura-Bédmar

Abstract: Improper use of synthetic medical texts created with generative artificial intelligence (AI) poses the risk of spread of fake content and disinformation. Detection of AI-generated text is a need for protecting non-specialist readers from manipulation. We introduce MedAID-ML, a multilingual dataset of biomedical texts for the detection of AI-generated text and authorship attribution. We gathered human-written texts in English, French, German and Spanish from authorized medical sources, and we generated artificial counterparts using three large language models: GPT-4o, MISTRAL and LLaMA-3.1. The current version includes 50% AI-generated and 50% human-written texts, and amounts to 13292 documents (3795449 tokens). We endeavoured to prevent data leakage by gathering a dedicated test set composed exclusively of texts published in 2025 and generated texts using model checkpoints from 2024. We present results from human evaluations and baseline experiments using a statistical classifier and state-of-the-art multilingual language models. By comparing performance metrics on test sets with and without potential data leakage, we provide evidence that prior work in this area might have reported inflated metric scores. In addition, we applied Integrated Gradients and SHAP to analyze model behavior to see whether classification decisions rely on linguistic markers specific to LLMs (https://github.com/Padraig20/MedAID-ML).

External IDs:doi:10.1007/978-3-032-04354-2_5