Probing the Uniquely Identifiable Linguistic Patterns of Conversational AI Agents

Anonymous

Abstract: Considerable effort has been dedicated to detecting machine-generated texts to prevent a situation where the widespread generation of text---with minimal cost and effort--- reduces trust in human interaction and factual information online. Our study takes a more refined approach by analysing different Conversational AI Agents (CAA). By constructing linguistic profiles for each AI agent, the aim is to identify the Uniquely Identifiable Linguistic Patterns (UILPs) for each model and to demonstrate the effectiveness of these UILPs in identifying their respective AI agents using authorship attribution techniques. Promisingly, we are able to classify AI agents based on their original texts with a weighted F1-score of 96.94%. Further, we can attribute AI agents according to their writing style (as specified by prompts), yielding a weighted F1-score of 95.84%, which sets the baseline for this task. By employing principal component analysis (PCA) for dimensionality reduction, we achieve a weighted F1-score ranging from 89.25% to 97.83%, and an overall weighted F1-score of 96.93%.

Paper Type: long

Research Area: Linguistic theories, Cognitive Modeling and Psycholinguistics

Contribution Types: Model analysis & interpretability

Languages Studied: English