Transfer Learning for Generalizable Automated LLM Improvement Pipeline for IVR Navigation

Vishal Sankar Ram, Jason Kushner, Manas Paldhe, Youngseo Son

Keywords: Prompt Optimization, Transfer Learning, Healthcare NLP applications

TL;DR: We introduce an automated prompt transfer pipeline for automating healthcare IVR navigation achieving 82% accuracy, surpassing an optimized human-designed prompt (79%) while reducing the prompt complexity by upt o 80%.

Abstract: Administrative tasks in the healthcare domain share linguistic commonalities, but it can be time-consuming to manually design LLM prompts for each use case. When calling health insurers, interactive voice response (IVR) systems cause delays in patient care and increase provider burnout due to complex routing and long hold times. Thus, IVR navigation models can offer significant time savings and reduce barriers to care. We propose a production-quality automated LLM pipeline which leverages a small number of human-labeled ground truth datasets to transfer specialized prompts from one task to another; specifically, we perform a cross-task transfer of our IVR navigation logic, adapting the prompt from reaching the claims department to reaching the patient benefit department. Our approach reduces prompt complexity by up to 80\% and obtains 82\% turn-level accuracy in real-world industrial healthcare settings, surpassing a human-designed prompt at 79\%.

Submission Type: Deployed

Submission Number: 470