Evaluating the Effectiveness of Large Language Models in Establishing Conversational Grounding

ACL ARR 2024 April Submission307 Authors

Abstract: Conversational grounding, vital for building dependable dialog systems, involves ensuring a mutual understanding of shared information. Despite its importance, there has been limited research on this aspect of conversation in recent years, especially after the advent of Large Language Models (LLMs). Previous studies, like those by Benotti and Blackburn(2021), highlighted the shortcomings of pre-trained language models in conversational grounding but lacked a standardized benchmark for comparison. This gap in research becomes more significant considering recent advances in language models, which have led to new emergent capabilities. In this paper, we aim to evaluate the performance of Large Language Models (LLMs) in various aspects of conversational grounding, analyze why some models perform better than others, and propose ways to enhance the capabilities of the models that lag behind. We demonstrate a direct correlation between the size of the pre-training data and conversational grounding abilities, meaning that they have independently acquired a specific form of pragmatic capabilities from larger pre-training datasets.

Paper Type: Long

Research Area: Dialogue and Interactive Systems

Research Area Keywords: Conversational Grounding, Dialog Systems

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 307