CADCon: An Approach for Robust Learning of Counterfactually Augmented Datasets based on Contrastive Learning

Anonymous

Abstract: During the fine-tuning process of Pre-trained Language Models (PLMs), they encounter relatively small datasets that may have spurious correlation patterns. Counterfactually Augmented Data (CAD) has emerged as a solution to make models less sensitive to such spurious patterns. While there has been progress in generating CAD due to advancements in generation models, the focus has primarily been on the quality of CAD, with limited attention given to training models for robustness. We introduce CADCon, a novel contrastive learning approach to enhance robustness by effectively utilizing CAD, rather than simply augmenting it. Firstly, we utilize an LLM-based generative model to generate counterfactual samples from original sentences. This is achieved by using a simple prompt, without human intervention or additional models. Secondly, we propose a tagging-based noise infusion method, which infuses noise into sentences without altering genuine tokens that have causal relationships with labels. Lastly, we perform contrastive learning so that counterfactual samples are distant from the original sentences and noise-infused samples are close. Our method effectively mitigates spurious correlations and improves robustness. We demonstrate that our method outperforms in both counterfactual task and domain generalization task.

Paper Type: long

Research Area: NLP Applications

Contribution Types: Model analysis & interpretability

Languages Studied: English