Adapting Blackbox Generative Models via Inversion

Sinjini Mitra, Rakshith Subramanyam, Rushil Anirudh, Jayaraman J. Thiagarajan, Ankita Shukla, Pavan K. Turaga

Keywords: Generative Modeling, Blackbox, Adaptation, Deployment

TL;DR: An alternative perspective for the problem of adapting large-scale generative models through a blackbox framework that is easy to deploy.

Abstract:

Adapting large-scale generative AI tools to differ-
ent end uses continues to be challenging, as many
industry grade image generator models are not
publicly available. Thus, to finetune an industry
grade image generator is not currently feasible
in the classical sense of finetuning certain layers
of a given deep-network. Instead, we present an
alternative perspective for the problem of adapt-
ing large-scale generative models that does not
require access to the full model. Recognizing
the expense of storing and fine-tuning generative
models, as well as the restricted access to weights
and gradients (often limited to API calls only), we
introduce AdvIN (Adapting via Inversion). This
approach advocates the use of inversion methods,
followed by training a latent generative model as
being equivalent to adaptation. We evaluate the
feasibility of such a framework on StyleGANs
with real distribution shifts, and outline some
open research questions. Even with simple in-
version and latent generation strategies, AdvIN
is surprisingly competitive to fine-tuning based
methods, making it a promising alternative for
end-to-end fine-tuning

Submission Number: 44