AMOR: A Recipe for Building Adaptable Modular Knowledge Agents Through Process Feedback

Anonymous

Abstract: The notable success of large language models (LLMs) has sparked an upsurge in building language agents to complete various complex tasks. We present A MOR, an agent framework based on open-source LLMs, which reasons with external knowledge bases and adapts to specific domains through human supervision to the reasoning process. A MOR builds reasoning logic over a finite state machine (FSM) that solves problems through autonomous executions and transitions over disentangled modules. This allows humans to provide direct feedback to the individual modules, and thus naturally forms process supervision. Based on this reasoning and feedback framework, we develop A MOR through two-stage fine-tuning: warm-up and adaptation. The former fine-tunes the LLM with examples automatically constructed from various public datasets and enables A MOR to generalize across different knowledge environments, while the latter tailors A MOR to specific domains using process feedback. Extensive experiments across multiple domains demonstrate the advantage of A MOR to strong baselines, thanks to its FSM-based reasoning and process feedback mechanism.

Paper Type: long

Research Area: Question Answering

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English