Collaborative QA using Interacting LLMs. Impact of Network Structure, Node Capability and Distributed Data.

TMLR Paper6543 Authors

Abstract: In this paper, we model and analyze how a network of interacting LLMs performs \textit{collaborative question-answering (CQA)} in order to estimate a ground truth given a distributed set of documents. This problem is interesting because LLMs often hallucinate when direct evidence to answer a question is lacking, and these effects become more pronounced in a network of interacting LLMs. The hallucination spreads, causing previously accurate LLMs to hallucinate. We study interacting LLMs and their hallucination by combining novel ideas of mean-field dynamics (MFD) from network science and the randomized utility model from economics to construct a useful generative model. We model the LLM with a latent state that indicates if it is truthful or not with respect to the ground truth, and extend a tractable analytical model considering an MFD to model the diffusion of information in a directed network of LLMs. To specify the probabilities that govern the dynamics of the MFD, we propose a randomized utility model. For a network of LLMs, where each LLM has two possible latent states, we posit sufficient conditions for the existence and uniqueness of a fixed point and analyze the behavior of the fixed point in terms of the incentive (e.g., test-time compute) given to individual LLMs. We experimentally study and analyze the behavior of a network of $100$ open-source LLMs with respect to data heterogeneity, node capability, network structure, and sensitivity to framing on multiple semi-synthetic datasets.

Submission Type: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

We have resolved the queries of reviewer uguw, NuyF and 2jo2 including:

1. Adding motivation for MFD and RUM model.

2. Redacting claims around the non-monotonicity of the population state with respect to the number of LLMs.

3. Making claims of diminishing returns exact.

4. Added explanation for how the theoretical results hold true for parallel interaction (used experimentally) versus the sequential interaction (in the system model).

5. Nit Smaller typos, clarifying notation, improving the figure captions and updating the figures to be consistent.

6. Added experiment results for the effect of increasing the number of LLMs.

Assigned Action Editor: ~George_Trimponias2

Submission Number: 6543