Evaluation and Continual Improvement for an Enterprise AI Assistant

Download PDF
Open Webpage

Akash Maharaj, Kun Qian, Uttaran Bhattacharya, Sally Fang, Horia Galatanu, Manas Garg, Rachel Hanessian, Nishant Kapoor, Ken Russell, Shivakumar Vaithyanathan, Yunyao Li

Published: 2024, Last Modified: 20 Nov 2025CoRR 2024EveryoneRevisionsBibTeXCC BY-SA 4.0
Abstract: The development of conversational AI assistants is an iterative process with multiple components. As such, the evaluation and continual improvement of these assistants is a complex and multifaceted problem. This paper introduces the challenges in evaluating and improving a generative AI assistant for enterprises, which is under active development, and how we address these challenges. We also share preliminary results and discuss lessons learned.