Ask the experts: sourcing a high-quality nutrition counseling dataset through Human-AI collaboration
Abstract: Recent publicly available Large Language Models (LLMs) are being employed by end-users for various tasks, including sensitive ones such as health counseling, disregarding potential safety concerns. It is thus necessary to understand how adequately LLMs perform in such domains. We conduct a case study on ChatGPT in nutrition counseling, a popular use-case where the model supports a user with their dietary struggles. We crowd-source real-world diet-related struggles, then work with nutrition experts to generate supportive text using ChatGPT. Finally, experts evaluate the safety and text quality of ChatGPT's output. The result is the HAI-coaching dataset, containing ~2.4K crowdsourced dietary struggles and ~97K corresponding ChatGPT-generated and expert-annotated supportive texts. We analyse ChatGPT's performance, discovering potentially harmful behaviours, especially for sensitive topics like mental health. Finally, we use HAI-coaching to test open LLMs on various downstream tasks, showing that even the latest models struggle to achieve good performance. HAI-coaching is available at https://anonymous.4open.science/r/3z2.
Paper Type: Long
Research Area: Resources and Evaluation
Research Area Keywords: LLMs,diet,nutrition,evaluation and metrics,human-in-the-loop,human evaluation,prompting,fine-tuning,automatic evaluation,healthcare applications,clinical NLP,corpus creation,benchmarking,evaluation methodologies,evaluation
Contribution Types: NLP engineering experiment, Data resources, Data analysis
Languages Studied: English
Submission Number: 5767
Loading