VerificAgent: Integrating Expert Knowledge and Fact-Checked Memory for Robust Domain-Specific Task Planning

Thong Q. Nguyen, Shubhang Desai, Yash Jain, Tanvir Aumi, Vishal Chowdhary

Submission Track: Paper Track (up to 8 pages)

Keywords: CUA, memory, continual learning, human in the loop

TL;DR: VerificAgent fuses a small expert‐seeded knowledge base with dynamically learned, fact‐checked memory to drive robust GUI task planning—doubling success rates on OSWorld Office tasks without any model fine‐tuning.

Abstract:

Continual memory augmentation allows computer-use agents (CUAs) to learn from past interactions and refine their task-solving strategies over time. However, unchecked memory accumulation can introduce spurious or hallucinated “learnings” that degrade agent performance—particularly in domain-specific workflows such as productivity software.
We present a novel framework, \textsc{VerificAgent}, that effectively manages memory for CUAs through (1) an expert-curated seed of domain knowledge, (2) iterative, trajectory-based memory refinement during training, and (3) a post-hoc fact-checking pass by human experts to sanitize accumulated memory before deployment.
On OSWorld productivity tasks, \textsc{VerificAgent} achieves a 111.1\% relative improvement in success rate over baseline CUA without any additional fine-tuning.

Submission Number: 35