Dually Self-Improved Counterfactual Data Augmentation Using Large Language Model

ACL ARR 2024 December Submission3 Authors

Abstract:

In this paper, we propose a novel self-improvement counterfactual data augmentation with large language model. First, attention mechanisms do provide a look into the inner workings of a model and capture meaningful correlations among intermediate states of input that can explain the model’s predictions. Therefore, we leverage the attention distribution of the model as the first self-improvement strategy to identify the top N casual terms, which is lightweight method not relying on external tools or models. Then prompt engineering with few shots is applied on a general LLM to produce a diverse set of perturbations for the dataset.
Further, to enhance the availability of the counterfactual data, a LLM self-thinking strategy is introduced as the second self-improvement strategy, where the LLM is optimized with the high-confidence examples that generated by itself through the prompt engineering. The self-improved LLM is treated as the final generator to produce CAD again. After filtering CAD and integrate them into original data, we retrain the baseline model with a balanced loss function and evaluate the performance.

Paper Type: Long

Research Area: Efficient/Low-Resource Methods for NLP

Research Area Keywords: counterfactual data augmentation, attention, large language model, self-improvement

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 3