DermAI: A Chatbot Assistant for Skin Lesion Diagnosis Using Vision and Large Language Models

Viet-Tham Huynh, Trong-Thuan Nguyen, Thao Thi Phuong Dao, Tam V. Nguyen, Minh-Triet Tran

Abstract: In dermatology, the demand for accurate skin lesion diagnoses is critical, especially during peak times like summer when skin cancer screenings surge. The need for efficient processing of large volumes of medical images and the risk of human error highlights the importance of innovative diagnostic tools. In this paper, we propose DermAI, an advanced AI-driven framework to improve diagnostic accuracy and efficiency in skin lesion analysis. Our DermAI framework combines a state-of-the-art segmentation model and a large language model to assist clinicians in interpreting medical images swiftly and precisely. Our framework isolates and analyzes key lesion features using advanced segmentation models and vision encoders, while a large language model provides contextual insights to understand lesion characteristics and potential malignancies. By integrating visual and linguistic analysis, our DermAI framework reduces diagnostic errors, alleviates clinician workloads, and enhances patient care with faster, more accurate results, supporting dermatologists in making informed decisions and advancing AI-assisted diagnostics.