The Effects of Data Augmentation on Confidence Estimation for LLMs

ACL ARR 2024 December Submission1504 Authors

Abstract: Confidence estimation is crucial for reflecting the reliability of large language models (LLMs), particularly in the widely used closed-source models. Current methods about confidence for black-box LLMs are susceptible to the overconfidence of LLMs, leading to suboptimal performance. To address this challenge, we study the impact of data augmentation on confidence estimation. Our findings indicate that data augmentation strategies can achieve better performance and mitigate the impact of overconfidence. We investigate the influential factors related to this and discover that, while preserving semantic information, greater data diversity enhances the effectiveness of augmentation. Furthermore, the impact of different augmentation strategies varies across different range of application. Considering parameter transferability and usability, the random combination of augmentations is a promising choice.

Paper Type: Short

Research Area: Special Theme (conference specific)

Research Area Keywords: Evaluation of Generalization Ability

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 1504