User intent driven retrieval augmented generation frameworks for auto-assisting compliance questionnaires

Dhaval Salwala, Seshu Tirupathi, Elizabeth M. Daly

Keywords: Intent to questionnaires, intent driven RAG, RAG-based LLM, human-in-the-loop

TL;DR: A one-shot human-in-the-loop RAG-based LLM approach to improve the accuracy of auto-assisting compliance questionnaires.

Abstract: AI models in production can pose risks related to ethics, regulations and compliance. Compliance frameworks and policies in organisations are fundamental in managing these risks. Questionnaires are an important tool adopted by organisations where owners or users of these models provide predefined information for review prior to deploying/using these AI models which can be mechanical and time-consuming. This paper discusses a retrieval augmented generation (RAG) framework to assist the end-user fill these questionnaires. In particular, early results show that one-shot human-in-the-loop RAG provides significant performance improvement in auto-assisting as compared to a traditional RAG model or a direct LLM model.

Submission Number: 3