Stylistic Shifts in Human–LLM Conversations: Challenges and Adaptation

Fulei Zhang, Zhou Yu

Keywords: Natural Language Processing, Conversational agents, Human-computer interaction

TL;DR: We empirically show users talk differently to LLMs than humans, and find that augmenting training data with stylistic rewrites improves model robustness more than inference-time query reformulation.

Abstract: As Large Language Models (LLMs) are increasingly deployed in customer-facing applications, a critical yet underexplored question is how users communicate differently with AI-driven chatbots compared to human associates. In this study, we present empirical evidence that users adopt distinct communication styles when interacting with chatbots versus human representatives. Our analysis reveals significant differences in grammatical fluency, politeness, and lexical diversity between the two settings. These findings suggest that models trained exclusively on human-human interaction data may not adequately accommodate the communication style shift that occurs once an LLM chatbot is deployed. To enhance LLM robustness to post-launch communication style changes, we experimented with two strategies: (1) data augmentation during the post-training phase and (2) inference-time user message reformulation. Our results indicate that models trained on stylistically diverse datasets significantly outperform those trained exclusively on original or stylistically uniform datasets, while inference-time reformulation proved less effective. These insights help us to better adapt our models for improved LLM-user interaction experiences.

Submission Number: 120