CAST: Sparse Fine-Tuning with Counterfactual Data Augmentation
Abstract: In the domain of transfer learning for pre-trained models, fine-tuning specific parameters rather than the entire model has become a prevalent trend. Sparse fine-tuning has proven effective. Counterfactual Data Augmentation have been shown to enhance the generalization ability of models. This study proposes a fine-tuning method that combines the advantages of both approaches, which is called "Counterfactual Augmented Sparse Tuning" (CAST). Inspired by the Lottery Ticket Hypothesis, this method identifies significant parameter changes by comparing models trained on counterfactual data with those trained on original data, thereby constructing a mask table for model parameters. To further enhance model sparsity, we introduce a counterfactual data impact factor, which adjusts the specific influence of counterfactual data on the model training outcomes. The CAST method achieved the best accuracy rates of 90.2\% and 76\% in counterfactual data augmentation tasks for sentiment analysis and natural language inference tasks. It was observed that CAST successfully resisted catastrophic shifts in dataset distribution. The CAST model not only improves performance in specific NLP tasks but also reduces the risk of data distribution shift and enhances the model's ability to capture key features.
Paper Type: Long
Research Area: Efficient/Low-Resource Methods for NLP
Research Area Keywords: Efficient/Low-Resource Methods for NLP；Machine Learning for NLP；Sentiment Analysis, Stylistic Analysis, and Argument Mining
Contribution Types: Approaches low compute settings-efficiency
Languages Studied: English
Submission Number: 1172
Loading