Usability, Engagement, and Report Usefulness of Chatbot-Based Family Health History Data Collection: Mixed Methods Analysis (Preprint)

Michelle Hoang Nguyen, João Sedoc, Casey Overby Taylor

Abstract: Background: Family health history (FHx) is an important predictor of a person’s genetic risk, but is not collected by many adults in the United States. Objective: The objective of this work was to test and compare the usability, engagement, and report usefulness of two Web-based methods to collect FHx. Methods: This mixed-methods study compared FHx data collection using a flow-based chatbot (KIT) and a form-based method. KITs’ design was optimized to reduce user burden. Two crowdsourced platforms were used to recruit and randomize individuals to one of the two FHx methods. All participants were asked to complete a questionnaire to assess the usability of the method, usefulness of a report summarizing their experience, chatbot desired enhancements, and general user experience. Engagement was studied using log data collected by the methods. We used qualitative findings from analyzing free-text comments to supplement the primary quantitative results. Results: Participants randomized to KIT reported higher usability than those randomized to the form, with a mean System Usability Scale (SUS) score of 80.2 in comparison to 61.9 (P < .001). Findings from analyzing engagement reflected design differences in onboarding process. KIT users spent less time entering FHx information and reported more conditions when compared to form users (5.90 vs 7.97 minutes on average, p = 0.036; and 7.8 vs 10.1 conditions, p 0.038). Both KIT and form users somewhat agreed that the report was useful (Likert scale ratings of 4.08 and 4.29, respectively). When asked about desired enhancements, personalization was the highest rated feature (92% rated medium to high priority). Qualitative analyses revealed positive and negative characteristics of both KIT and the form-based method. With a focus on findings from respondents randomized to KIT, most indicated it was easy to use, easy to navigate and that they were able to respond to and understand user prompts. Negative comments collected were about KITs personality, conversational pace, and ability to manage errors. For both KIT and form respondents, qualitative results revealed common themes including a desire for more information about conditions and a mutual appreciation for the multiple-choice button response format. There was also a desire to report more health information than what KIT prompted (e.g., personal health history) and for KIT to provide more personalized responses. Conclusions: We showed that KIT provided a usable way to collect FHx history. We also identified design considerations to improve chatbot-based FHx data collection. First, the final report summarizing the FHx collection experience should be improved to provide more value for patients. Second, the onboarding chatbot prompt may impact data quality and should be carefully considered. Last, we highlighted several areas that can be improved with a move from a flow-based chatbot to a large language model implementation strategy.

External IDs:doi:10.2196/preprints.55164