Show or Tell? Modeling the evolution of request-making in Human-LLM conversations

ACL ARR 2025 July Submission1206 Authors

Abstract:

Chat logs provide a rich source of information about LLM users, but patterns of user behavior are often masked by the variability of queries. We present a new task, segmenting chat queries into contents of requests, roles, query-specific context, and additional expressions.
We find that, despite the familiarity of chat-based interaction, request-making in LLM queries remains significantly different from comparable human-human interactions.
With the data resource, we introduce an important perspective of diachronic analyses with user expressions.
We find that query patterns vary between early ones emphasizing requests, and individual users explore patterns but tend to converge with experience.
Finally, we show that model capabilities affect user behavior, particularly with the introduction of new models, which are traceable at the community level.

Paper Type: Long

Research Area: Human-Centered NLP

Research Area Keywords: human-AI interaction/cooperation,human-centered evaluation,user-centered design,participatory/community-based NLP

Contribution Types: NLP engineering experiment, Publicly available software and/or pre-trained models, Data resources, Data analysis, Theory

Languages Studied: English

Submission Number: 1206