An In-depth Investigation of User Response Simulation for Conversational Search

Zhenduo Wang, Zhichao Xu, Vivek Srikumar, Qingyao Ai

Keywords: conversational search, user response simulation, natural language generation

Abstract:

Conversational search has seen increased recent attention in both the IR and NLP communities.
It seeks to clarify and solve users' search needs through multi-turn natural language interactions.
However, most existing systems are trained and demonstrated with recorded or artificial conversation logs.
Eventually, conversational search systems should be trained, evaluated, and deployed in an open-ended setting with unseen conversation trajectories.
A key challenge is that training and evaluating such systems both require a human-in-the-loop, which is expensive and does not scale.
One strategy is to simulate users, thereby reducing the scaling costs.
However, current user simulators are either limited to only responding to yes-no questions from the conversational search system or unable to produce high-quality responses in general.

In this paper, we show that existing user simulation systems could be significantly improved by a smaller finetuned natural language generation model.
However, rather than merely reporting it as the new state-of-the-art, we consider it a strong baseline and present an in-depth investigation of simulating user response for conversational search.
Our goal is to supplement existing work with an insightful hand-analysis of unsolved challenges by the baseline and propose our solutions.
The challenges we identified include (1) a blind spot that is difficult for the model to learn, and (2) a specific type of misevaluation in the standard empirical setup.
We propose a new generation system to effectively cover the training blind spot and suggest a new evaluation setup to avoid misevaluation.
Our proposed generation system leads to significant improvements over existing systems and large language models such as GPT-4.
Besides number improvements, we also provide insights into the nature of user simulation to facilitate future work.

Track: Search

Submission Guidelines Scope: Yes

Submission Guidelines Blind: Yes

Submission Guidelines Format: Yes

Submission Guidelines Limit: Yes

Submission Guidelines Authorship: Yes

Student Author: Yes

Submission Number: 728