Mixture Weight Estimation and Model Prediction in Multi-source Multi-target Domain Adaptation

Yuyang Deng, Ilja Kuzborskij, Mehrdad Mahdavi

Keywords: Multi-source domain adaptation; minimax optimization; learning theory

Abstract:

We consider a problem of learning a model from multiple sources with the goal to perform
well on a new target distribution. Such problem arises in
learning with data collected from multiple sources (e.g. crowdsourcing) or
learning in distributed systems, where the data can be highly heterogeneous. The
goal of learner is to mix these data sources in a target-distribution aware way and
simultaneously minimize the empirical risk on the mixed source. The literature has made some tangible advancements in establishing
theory of learning on mixture domain. However, there are still two unsolved problems. Firstly, how to estimate the optimal mixture of sources, given a target domain; Secondly, when there are numerous target domains, we have to solve empirical risk minimization for each target on possibly unique mixed source data , which is computationally expensive. In this paper we address both problems efficiently and with guarantees.
We cast the first problem, mixture weight estimation as convex-nonconcave compositional minimax, and propose an efficient stochastic
algorithm with provable stationarity guarantees.
Next, for the second problem, we identify that for certain regime,
solving ERM for each target domain individually can be avoided, and instead parameters for a target optimal
model can be viewed as a non-linear function on
a space of the mixture coefficients.
To this end, we show that in offline setting, a GD-trained overparameterized neural network can provably learn such function.
Finally, we also consider an online setting and propose an label efficient online algorithm, which predicts parameters for new models given arbitrary sequence of mixing coefficients, while enjoying optimal regret.

Submission Number: 7714