Flash-DD: An Ultra Parameter-Efficient Approach to Dataset Distillation

Ruonan Yu, Songhua Liu, Xinchao Wang

Keywords: Dataset Distillation, Dataset Condensation, Efficient Learning

TL;DR: We propose a DD-oriented model parameter reduction method that automatically determines the optimal capacity of teacher models and eliminates redundant parameters for dataset distillation tasks.

Abstract: Dataset distillation (DD) aims to create a smaller dataset that encapsulates the essential knowledge of a larger dataset, thereby reducing storage demands and accelerating downstream training. For large-scale dataset distillation, state-of-the-art methods achieve satisfactory performance by using soft labels generated by well-trained teacher models during downstream training. However, it will cause some issues: (1) a substantial amount of additional storage is required to retain the teacher models, often significantly exceeding the storage needed for the synthetic images; (2) generating labels through these teacher models slows down the downstream training process, counteracting the efficiency goals of dataset distillation; and (3) downstream training guided by these teacher models, according to our studies, yields suboptimal performance. Focusing on these drawbacks, in this paper, we propose plug-and-play parameter-efficient label generation techniques for dataset distillation, which maximizes the benefits of limited model parameters and can be generalized to different DD methods, datasets, and settings. Specifically, we propose a DD-oriented model parameter reduction method that automatically determines the optimal capacity of teacher models and eliminates redundant parameters for dataset distillation tasks. Furthermore, for additional parameter space, we turn to model ensemble strategies and propose guidelines to optimize the utilization efficiency of the additional space. Compared to the state-of-the-art methods, Flash-DD requires only 0.03% of the additional storage and significantly accelerates downstream label generation by 843.81x while maintaining comparable performance. Alternatively, with a mere 1.8% storage budget, it can boost accuracy by up to 13.4% over previous leading methods. Our code will be available.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 8479