CompactDP: Category-Aware Feature Compactness for Differential Privacy

Shilin Zhang

Keywords: Data Privacy, DP-SGD, Feature Space Compactness, Membership Inference Attack

TL;DR: CompactDP targets the root cause of leaks: sparse, high-dimensional features. Category-aware feature contraction that tightens scattered data distributions where individual training samples get memorized.

Abstract: The rapid growth of AI models raises critical privacy concerns due to their tendency to memorize training data, making them vulnerable to extraction and membership inference attacks (MIAs). Traditional privacy-preserving methods like DP-SGD often degrade model utility, limiting their applicability in sensitive fields. We observe that compact class-wise feature manifold embeddings inherently reduce privacy risks by smoothing probability density functions (PDFs), which diminishes the influence of individual training samples and lowers memorization. Leveraging this insight, we propose \textit{Class-wise Compactness for Privacy} (C4P) method, a noise-free feature contraction framework that directly addresses the root cause of privacy leakage, sparse, high-dimensional features, via feature contraction rather than relying on gradient noise. C4P can be trained independently and achieves a superior privacy-utility trade-off, with empirical privacy guarantee comparable to DP-SGD ($\epsilon=1$). C4P attains 95.82\% accuracy while limiting MIA risk comparable with DP-SGD ($\epsilon=1$) on CIFAR10. Notably, leveraging the optimized feature embedding, DP-SGD maintains robust model utility while preserving rigorous privacy guarantees across varying privacy budgets. Extensive experiments on FashionMNIST and MedicalMNIST further validate its favorable utility-privacy trade-off across diverse metrics.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 14605