Go to ACL ARR 2026 January homepageA Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agents
Keywords: LLM stylistic control, side effect, prompt; steering
Abstract: Style features such as friendly, helpful, and concise are widely used in prompts to steer the behavior of Large Language Model (LLM) conversational agents, yet their unintended side effects remain poorly understood. In this work, we present the first systematic study of cross-feature stylistic side effects by surveying 127 conversational agent papers from the ACL Anthology and identifying 12 commonly used style features. Using controlled synthetic dialogues across both task-oriented and open-domain settings, we quantify how prompting for one style feature causally affects others through a pairwise LLM-as-a-Judge evaluation framework. Our results reveal consistent and structured side effects—for example, prompting for conciseness significantly reduces perceived expertise—demonstrating that stylistic features are deeply entangled rather than orthogonal. To support future research, we introduce CASSE (Conversationalational Agent Stylistic Side Effects), a dataset capturing these complex interactions. We further evaluate prompt-based and activation-steering–based mitigation strategies and find that while they can partially restore suppressed traits, they often degrade the primary intended style, challenging assumptions of faithful style control in LLMs and highlighting the need for multi-objective, principled approaches to safe and targeted stylistic steering.
Paper Type: Long
Research Area: Dialogue and Interactive Systems
Research Area Keywords: evaluation and metrics, task-oriented, bias/toxicity, conversational modeling
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data resources, Data analysis, Surveys
Languages Studied: English
Submission Number: 10182
Loading