PRDetect: Perturbation-Robust LLM-generated Text Detection Based on Syntax Tree
Abstract: As LLM-generated text becomes increasingly prevalent on the internet, which may contain hallucinations or biases, detecting such content has emerged as a critical area of research. Recent methods have demonstrated impressive performance in detecting text generated entirely by LLMs. However, in real-world scenarios, users often make perturbations to the LLM-generated text, and the robustness of existing detection methods to these perturbations has not been sufficiently explored. This paper empirically investigates this question and finds that even minor perturbation can severely degrade the performance of current detection methods. To address this issue, we find that the syntactic tree is minimally affected by disturbances and exhibits differences between human-written text and LLM-generated text. Therefore, we propose a detection method based on syntactic trees, which can capture features invariant under perturbations. It demonstrate significantly improved robustness against perturbation on the HC3 and GPT-3.5-mixed datasets.
Paper Type: Short
Research Area: Generation
Research Area Keywords: analysis
Contribution Types: Data analysis
Languages Studied: English
Submission Number: 4123
Loading