Toward Bit-Efficient Dataset Condensation: A General Framework

ICLR 2026 Conference Submission12801 Authors

Keywords: Dataset Condensation

Abstract:

Dataset condensation aims to distill a large-scale dataset into a compact set of synthetic samples for efficient training. Existing methods primarily focus on reducing the number of samples but generally assume full-precision representations. While effective, this assumption limits their applicability in resource-constrained scenarios due to several major drawbacks:
(1) \textit{Transmission bottlenecks}—full-precision datasets consume excessive bandwidth and introduce latency during network transfer, especially in cloud–edge collaborative learning;
(2) \textit{Memory overhead}—storing and processing full-precision data rapidly exhausts GPU memory or RAM, restricting batch sizes; and
(3) \textit{Hardware underutilization}—modern accelerators are optimized for low-precision operations, yet full-precision data prevents full efficiency gains in training and inference.
To address these challenges, we propose a novel approach that fine-tunes distilled full-precision datasets into compact low-bit representations, substantially reducing memory usage with minimal computational overhead. Central to our method is a differentiable bit-conscious optimization framework. This framework allows more synthetic samples to be stored within the same memory budget, thereby improving downstream performance.
Beyond the algorithmic contribution, we provide theoretical analysis that characterizes (1) the trade-off between compression error and generalization error under memory constraints, and (2) the extent to which Fisher information is preserved under bit compression. Extensive experiments on multiple benchmarks against state-of-the-art baselines validate both the effectiveness and efficiency of our method.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 12801