Causal reasoning for controllable generative modeling

Anonymous

Keywords: Controllable generation, causal discovery

TL;DR: We propose a method to infer implicit causal structure over generative models and show that knowledge of this is beneficial for controllable generation.

Abstract:

Deep generative models excel at generating complex, high-dimensional data, often exhibiting impressive generalization beyond the training distribution. However, many such models in use today are black-boxes trained on large unlabelled datasets with statistical objectives and lack an interpretable understanding of the latent space required for controlling the generative process.
We propose CAGE, a framework for controllable generation in latent variable models based on casual reasoning.
Given a pair of attributes, CAGE infers the cause-effect relationships between these attributes by estimating unit-level causal effects. % in the latent space of the generative model. We design a geometric procedure for estimating these effects that applies broadly to any latent variable model. Thereafter, we use the inferred cause-effect relationships to design a novel strategy for controllable generation based on counterfactual sampling. Through a series of large-scale human evaluations, we demonstrate that generating counterfactual samples which leverage the underlying causal relationships inferred via CAGE leads to subjectively more realistic images.