Can LLMs Contribute to Cooperative Fact-Checking? A Field Evaluation on X Community Notes

Haiwen Li, Michiel A. Bakker

Keywords: Misinformation, Fact-Checking, Large Language Models, Community Notes, Cooperative AI

TL;DR: This work provides field evidence that LLMs can generate content accepted across diverse viewpoints in a crowdsourced fact-checking system, demonstrating a form of AI contribution to cooperative human systems.

Abstract: Combating misinformation on social media increasingly relies on collective, user-driven fact-checking. X Community Notes exemplifies this approach: users with different viewpoints propose contextual notes to misleading content and evaluate them, and a bridging algorithm surfaces notes that achieve cross-partisan agreement. We study whether large language models (LLMs) can effectively participate in this process. We present the first field evaluation of LLM fact-check writing on a live platform, using X Community Notes' "AI writer" feature. Over a three-month period, our LLM system wrote 1,614 notes on 1,597 tweets, alongside 1,332 human-written notes on the same tweets, evaluated using 108,169 ratings from 42,521 users. At the rating level, LLM notes receive more positive evaluations than human-written notes across raters with different political leanings. Although platform constraints limit the exposure of LLM notes relative to human notes, note-level analysis that accounts for the differential exposure confirms the same advantage. Together, these findings show that LLMs can generate broadly helpful fact-checks at scale in real-world settings, and provide field evidence that LLMs can contribute to a cooperative information system where success depends on acceptance across disagreement.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 152