BEARD: Benchmarking the Adversarial Robustness for Dataset Distillation

ICLR 2026 Conference Submission14733 Authors

Keywords: Dataset distillation, Benchmark, Adversarial robustness, Adversarial robustness distillation, Adversarial game framework

TL;DR: We introduce a benchmark to evaluate the adversarial robustness of models trained using dataset distillation methods, facilitating a standardized assessment of security concerns in this area.

Abstract: Dataset Distillation (DD) compresses large-scale datasets into smaller synthesized datasets, enabling efficient model training while preserving high test performance. However, existing DD methods primarily focus on accuracy and largely neglect adversarial robustness, potentially exposing models to security risks in critical applications. Evaluating robustness is therefore essential but remains challenging due to complex interactions among distillation methods, model architectures, and attack strategies. Moreover, current benchmarks provide only partial coverage and lack a unified perspective in the DD domain. To address this gap, we introduce BEARD, an open and unified benchmark for systematically evaluating the adversarial robustness of models trained on distilled datasets from representative DD methods such as DM, IDM, and BACON. BEARD supports diverse adversarial attacks, including FGSM, PGD, and C&W, and widely used datasets such as CIFAR-10/100 and TinyImageNet. Using an adversarial game framework, we define three key metrics: Robustness Ratio (RR), Attack Efficiency Ratio (AE), and Comprehensive Robustness-Efficiency Index (CREI). We conduct systematic evaluations and analyses across unified benchmarks, varying images-per-class (IPC) settings and adversarial training strategies, showing that dataset distillation consistently enhances adversarial robustness, with adversarial training providing further improvements. The leaderboard is available at https://beard-leaderboard.github.io/, along with a library of model and dataset pools to support reproducible research. Code is accessible at https://anonymous.4open.science/r/BEARD-6B8A/.

Primary Area: datasets and benchmarks

Submission Number: 14733