Large Language Models can be Guided to Evade AI-generated Text Detection

Ning Lu, Shengcai Liu, Rui He, Yew-Soon Ong, Qi Wang, Ke Tang

Keywords: AI-content detection, in-context learning

TL;DR: We propose a novel substitution-based in-context example optimization algorithm to automatically generate prompts which make LLMs evade AI-generated text detectors.

Abstract:

Large language models (LLMs) have shown remarkable performance in various tasks and have been extensively utilized by the public.
However, the increasing concerns regarding the misuse of LLMs, such as plagiarism and spamming, have led to the development of multiple detectors, including fine-tuned classifiers and statistical methods.
In this study, we equip LLMs with prompts, rather than relying on an external paraphraser, to evaluate the vulnerability of these detectors.
We propose a novel Substitution-based In-Context example Optimization method (SICO) to automatically construct prompts for evading the detectors.
SICO is cost-efficient as it requires only 40 human-written examples and a limited number of LLM inferences to generate a prompt.
Moreover, once a task-specific prompt has been constructed, it can be universally used against a wide range of detectors.
Extensive experiments across three real-world tasks demonstrate that SICO significantly outperforms the paraphraser baselines
and enables GPT-3.5 to successfully evade six detectors, decreasing their AUC by 0.5 on average.
Furthermore, a comprehensive human evaluation as well as a validation experiment in the wild show that the SICO-generated text achieves human-level readability and task completion rates.
Finally, the strong performance of SICO exhibits its potential as a reliable evaluation tool for future detectors.

Primary Area: societal considerations including fairness, safety, privacy

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3068