Multiobjective distribution matching

Xiaoyuan Zhang, Peijie Li, Yingying Yu, Yichi Zhang, Han Zhao, Qingfu Zhang

TL;DR: Discuss several issues in generating a distribution which resembles multiple distributions at the same time.

Abstract:

Distribution matching is a key technique in machine learning, with applications in generative models, domain adaptation, and algorithmic fairness. A related but less explored challenge is generating a distribution that aligns with multiple underlying distributions, often with conflicting objectives, known as a Pareto optimal distribution.
In this paper, we develop a general theory based on information geometry to construct the Pareto set and front for the entire exponential family under KL and inverse KL divergences. This formulation allows explicit derivation of the Pareto set and front for multivariate normal distributions, enabling applications like multiobjective variational autoencoders (MOVAEs) to generate interpolated image distributions.
Experimental results on real-world images demonstrate that both algorithms can generate high-quality interpolated images across multiple distributions.

Lay Summary: We study how to generate a distribution, which resembles to multiple distributions at the same time in this paper. We formulate this problem as an optimization problem defined in the dually flat manifold. In this way, we could derive the explicit formulation for the PS for the exponational family. Multiobjective distribution matching is a fundamental research problem in machine learning, which is highly related with group DRO, algorithmic fairness, and generative models.

Primary Area: Deep Learning

Keywords: multiobjective optimization; multitask learning; distribution matching

Submission Number: 2054