Go to ACL ARR 2025 February homepageLM Agents for Coordinating Multi-User Information Gathering
Abstract: This paper introduces PeopleJoin, a benchmark for evaluating LM-mediated collaborative problem solving. Given a user request, PeopleJoin agents must identify teammates who might be able to assist, converse with these teammates to gather information, and finally compile a useful answer or summary for the original user. PeopleJoin comprises two evaluation domains: PeopleJoin-QA, focused on questions about tabular data, and PeopleJoin-DocCreation, focused on document creation tasks. The two domains are adapted from existing NLP benchmarks for database question answering and multi-document summarization; here, however, the information needed to complete these tasks is distributed across synthetic ``organizations'' of 2--20 users, simulating natural multi-user collaboration scenarios. We implemented several popular LM agent architectures, evaluating their accuracy and efficiency at completing tasks, and highlight new research questions that can be studied using PeopleJoin.
Paper Type: Long
Research Area: Dialogue and Interactive Systems
Research Area Keywords: task-oriented, evaluation and metrics, knowledge augmented, applications
Contribution Types: NLP engineering experiment, Data resources
Languages Studied: English
Submission Number: 2521
Loading