ROME is Forged in Adversity: Robust Distilled Datasets via Information Bottleneck

Zheng Zhou, Wenquan Feng, Qiaosheng Zhang, Shuchang Lyu, Qi Zhao, Guangliang Cheng

TL;DR: We introduce ROME, a method that enhances the adversarial robustness of dataset distillation by leveraging the information bottleneck principle, leading to significant improvements in robustness against both white-box and black-box attacks.

Abstract: Dataset Distillation (DD) compresses large datasets into smaller, synthetic subsets, enabling models trained on them to achieve performance comparable to those trained on the full data. However, these models remain vulnerable to adversarial attacks, limiting their use in safety-critical applications. While adversarial robustness has been extensively studied in related fields, research on improving DD robustness is still limited. To address this, we propose ROME, a novel method that enhances the adversarial RObustness of DD by leveraging the InforMation BottlenEck (IB) principle. ROME includes two components: a performance-aligned term to preserve accuracy and a robustness-aligned term to improve robustness by aligning feature distributions between synthetic and perturbed images. Furthermore, we introduce the Improved Robustness Ratio (I-RR), a refined metric to better evaluate DD robustness. Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate that ROME outperforms existing DD methods in adversarial robustness, achieving maximum I-RR improvements of nearly 40% under white-box attacks and nearly 35% under black-box attacks. Our code is available at https://github.com/zhouzhengqd/ROME.

Lay Summary: Training modern machine learning models often requires large datasets, which can be expensive and difficult to collect. A popular solution is dataset distillation, where small synthetic datasets are created to help models learn efficiently. However, models trained on these synthetic datasets are often vulnerable to small but harmful changes in input data, known as adversarial attacks. This research presents ROME, a method that improves the reliability of models trained on distilled data. ROME uses ideas from information theory to keep the most useful parts of the data while reducing irrelevant noise. This allows the model to stay accurate even when the inputs are slightly changed in a malicious way. By improving both efficiency and robustness, ROME helps make machine learning systems more practical and secure in real-world scenarios such as autonomous driving and medical diagnosis.

Link To Code: https://github.com/zhouzhengqd/ROME

Primary Area: Deep Learning->Robustness

Keywords: Dataset distillation, adversarial robustness, information bottleneck, synthetic subsets, adversarial attacks

Submission Number: 1422