Aggregating Data for Optimal and Private Learning

Sushant Agarwal, Yukti Makhija, Rishi Saket, Aravindan Raghuveer

Keywords: Multiple Instance Regression, Learning from Label Proportions, k-means clustering, label-differential privacy, linear regression

Abstract: Multiple Instance Regression (MIR) and Learning from Label Proportions (LLP) are learning frameworks arising in many applications, where the training data is partitioned into disjoint sets or \emph{bags}, and only an aggregate label i.e., \emph{bag-label} for each bag is available to the learner. In the case of MIR, the bag-label is the label of an undisclosed instance from the bag, while in LLP, the bag-label is the mean of the bag's labels. In this paper, we study for various loss functions in MIR and LLP, what is the optimal way to partition the dataset into bags such that the utility for downstream tasks like linear regression is maximized. We theoretically provide utility guarantees, and show that in each case, the optimal bagging strategy (approximately) reduces to finding an optimal clustering of the feature vectors or the labels with respect to natural objectives such as $k$-means. We also show that our bagging mechanisms can be made \emph{label-differentially private}, incurring an additional utility error. We then generalize our results to the setting of Generalized Linear Models (GLMs). Finally, we experimentally validate our theoretical results.

Submission Number: 103