Coding Reliable LLM-based Integrated Task and Knowledge Agents with GenieWorksheets
Keywords: conversational agents, framework, task-oriented dialog agents, task and knowledge agents
TL;DR: A programmable framework for creating reliable LLM-based task-oriented agents that integrate various knowledge sources.
Abstract: Large Language Models (LLMs) present an opportunity to create automated assistants that can help users navigate complex tasks. However, existing approaches have limitations in handling conditional logic, integrating knowledge sources, and consistently following instructions. Researchers and industry professionals often employ ad hoc pipelines to construct conversational agents. These pipelines aim to maintain context, address failure cases, and minimize hallucinations, yet frequently fail to achieve these objectives. To this end, we present Genie – a programmable framework for creating task-oriented conversational agents that are designed to handle complex user interactions and knowledge queries. Unlike LLMs, Genie provides reliable grounded responses, with controllable agent policies through its expressive specification, Genie Worksheet. In contrast to dialog trees, it is resilient to diverse user queries, helpful with knowledge sources, and offers ease of programming policies through its declarative paradigm. The agents built using Genie outperforms the state-of-the-art method on complex logic domains in STARV2 dataset by up to 20.5%. Through a real-user study involving 62 participants, we show that Genie beats the GPT-4 Turbo with function calling baseline by 21.1%, 20.1%, and 61% on execution accuracy, dialogue act accuracy, and goal completion rate, respectively, on three diverse real-world domains.
Supplementary Material: zip
Primary Area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 10652
Loading