Bayesian Power Steering: An Effective Approach for Domain Adaptation of Diffusion Models

Jian Huang

Abstract:

We propose a Bayesian framework for fine-tuning
large diffusion models with a novel network structure called Bayesian Power Steering (BPS) .
We clarify the meaning behind adaptation from a
large probability space to a small probability
space and explore the task of fine-tuning pretrained models using learnable modules from a
Bayesian perspective. BPS extracts task-specific
knowledge from a pre-trained model’s learned
prior distribution. It efficiently leverages large diffusion models, differentially intervening different
hidden features with a head-heavy and foot-light
configuration. Experiments highlight the superiority of BPS over contemporary methods across a
range of tasks even with limited amount of data.
Notably, BPS attains an FID score of 10.49 under
the sketch condition on the COCO17 dataset