On Dataset Distillation with Two-fold Pseudo-Distribution Matching

Yutao Liu, Xiaobo Zhang, Hao Wang, Yan Yang, Wei Wang

Keywords: Dataset Distillation, Distribution Matching, Capacity Matching, Pseudo-Trajectory Matching, Pseudo-Label Matching

TL;DR: PTLM enhances dataset distillation via Pseudo-Trajectory Matching and Pseudo-Label Matching to resolve feature and distribution shifts in distribution matching.

Abstract: The goal of dataset distillation (DD) is to learn a compact synthetic dataset that maintains comparable generalization performance with the original. Distribution matching (DM), a leading DD approach, excels in addressing model scalability. However, current methods struggle with inherent feature and distribution shifts, facing a trade-off between efficiency and effectiveness. This paper reveals that in DM, models should prioritize similar samples (similar samples) when the image-per-class (IPC) is low, while incorporating diverse samples (diverse samples) as IPC increases to capture broader information. We call such a finding as \textit{capacity matching}, and then we propose a Two-fold Pseudo-Distribution matching (namely \underline{\textbf{P}}seudo-\underline{\textbf{T}}rajectory matching and Pseudo-\underline{\textbf{L}}abel \underline{\textbf{M}}atching (PTLM)) to address feature and distribution shifting issues in DM. Specifically, we design (1) a Pseudo-Trajectory Matching (PTM) to address feature shift, and (2) Pseudo-Label Matching (PLM) to address distribution shift. Our proposal is a plug-and-play component for any DM-based method. Experimental results on multiple real-world datasets show the efficiency and effectiveness of the proposed method. The source code is available at~\href{https://anonymous.4open.science/r/PTLM}{https://anonymous.4open.science/r/PTLM}.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 23373