On the Use of Mixup Augmentation for Synthetic Image Attribution in the Wild

Jun Wang, Benedetta Tondi, Mauro Barni

Abstract: Due to the continuous proliferation of generative models and the often unauthorized dissemination of synthetic images, the problem of tracing back the origin of a synthetic image, namely, synthetic image attribution, has raised a lot of interest. Despite the recent advances in this field, performing synthetic image attribution in the wild is still challenging. Recent literature on Open Set Recognition (OSR) in machine learning has shown that training augmentation has a strong effect on the open-set performance of OSR methods. Inspired by this literature, in this paper, we assess the impact on the synthetic image attribution task of a new type of augmentation, called mixup augmentation, that consists of interpolating data from different classes. Our experiments with models based on Convolutional Neural Networks (CNNs) as well as Vision Transformers (ViTs), reveal that this type of augmentation is indeed effective in improving robustness and generalization of synthetic image attribution classifiers, yet at the expense of a reduction of open-set performance. We also proposed a new variant of this type of augmentation, performing mixup of the samples with frozen labels, that keeps the benefits of the standard mixup in terms of robustness and generalization, at the same time mitigating the performance drop in open-set.