Few shot chain-of-thought driven reasoning to prompt LLMs for open ended medical question answering
Abstract: In this paper, we propose a modified version of the MedQA-USMLE dataset (MEDQANOOPT), which contains (i) open-ended medical questions without options, to mimic clinical scenarios along with (ii) clinician-approved reasoned answers. Further, we implement a prompt, riven by Chain of Thought (CoT) reasoning (MEDCODEX) to mirror the prospective process of (incremental reasoning) reaching a correct response to the medical questions. We empirically demonstrate how MEDCODEX outperforms the state-of-the-art 5-shot-codex-CoT-prompt (Liévin et al., 2022). We also present an approach that mirrors real-life clinical practice by first exploring multiple differential diagnoses (MEDCODEXFEWSHOTPROMPT) and subsequently narrowing down to a final diagnosis (CODEXFEWSHOTPROMPT). Finally, keeping in mind the importance of response verification in the medical setting, we utilize a reward model mechanism replacing the elimination performed by CODEXFEWSHOTPROMPT.
Paper Type: Long
Research Area: NLP Applications
Research Area Keywords: Generation, Human-Centered NLP, Interpretability and Analysis of Models for NLP, Language Modeling, Machine Learning for NLP, NLP Applications, Question Answering, Resources and Evaluation
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data resources, Data analysis
Languages Studied: English
Section 2 Permission To Publish Peer Reviewers Content Agreement: Authors grant permission for ACL to publish peer reviewers' content
Submission Number: 709
Loading