LONGAGENT: Achieving Question Answering for 128k-Token-Long Documents through Multi-Agent Collaboration
Abstract: Large language models (LLMs) have achieved tremendous success in understanding language and processing text.
However, question-answering (QA) on lengthy documents faces challenges of resource constraints and a high propensity for errors, even for the most advanced models such as GPT-4 and Claude2.
In this paper, we introduce \textsc{LongAgent}, a multi-agent collaboration method that enables efficient and effective QA over $128k$-token-long documents. \textsc{LongAgent} adopts a \textit{divide-and-conquer} strategy, breaking down lengthy documents into shorter, more manageable text chunks.
A leader agent comprehends the user's query and organizes the member agents to read their assigned chunks, reasoning a final answer through multiple rounds of discussion.
Due to members' hallucinations, it's difficult to guarantee that every response provided by each member is accurate.
To address this, we develop an \textit{inter-member communication} mechanism that facilitates information sharing, allowing for the detection and mitigation of hallucinatory responses.
Experimental results show that a LLaMA-2 7B driven by \textsc{LongAgent} can effectively support QA over $128k$-token documents, achieving $16.42\%$ and $1.63\%$ accuracy gains over GPT-4 on single-hop and multi-hop QA settings, respectively.
Paper Type: Long
Research Area: Question Answering
Research Area Keywords: Question Answering; Resources and Evaluation
Contribution Types: NLP engineering experiment, Data resources
Languages Studied: English
Submission Number: 4405
Loading