Who SAID that? Benchmarking Social Media AI Detection
Keywords: AI-generated text detection, benchmark, large language model
Abstract: AI-generated content (AIGC) has proliferated across various online platforms, offering both transformative prospects and posing significant risks related to misinformation and manipulation. Addressing these challenges, this paper introduces SAID (\underline{s}ocial media \underline{AI} \underline{d}etection), a novel benchmark developed to assess AIGC detection models' capabilities in realistic scenarios. It incorporates real AIGC data from popular social media platforms like Zhihu and Quora. Unlike existing benchmarks, SAID deals with content that reflects the sophisticated strategies employed by AIGCs on the Internet to evade detection or gain visibility, providing a more realistic and challenging evaluation landscape. A notable finding of our study, based on the Zhihu dataset, reveals that expert annotators can distinguish between AI-generated and human-generated texts with an average accuracy rate of 97\%. This finding necessitates a re-evaluation of human capability in recognizing AIGC in today’s widely AI-influenced environment. Furthermore, we present a new user-oriented AIGC detection challenge focusing on the practicality and effectiveness of identifying AIGC based on user information and answer content.
Primary Area: datasets and benchmarks
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6927
Loading