Few-shot Image Generation via Adaptation-Aware Kernel Modulation

Yunqing ZHAO, Keshigeyan Chandrasegaran, Milad Abdollahzadeh, Ngai-man Cheung

Keywords: Few-shot Image Generation, Modulation of Deep Neural Networks, Transfer Learning, Knowledge Preserve

TL;DR: We uncover some issues of state-of-the-art algorithms on few-shot image generation and propose a novel adaptation-aware kernel modulation method to improve few-shot image generation in different setups.

Abstract: Few-shot image generation (FSIG) aims to learn to generate new and diverse samples given an extremely limited number of samples from a domain, e.g., 10 training samples. Recent work has addressed the problem using transfer learning approach, leveraging a GAN pretrained on a large-scale source domain dataset and adapting that model to the target domain based on very limited target domain samples. Central to recent FSIG methods are knowledge preserving criteria, which aim to select a subset of source model's knowledge to be preserved into the adapted model. However, a major limitation of existing methods is that their knowledge preserving criteria consider only source domain/source task, and they fail to consider target domain/adaptation task in selecting source model's knowledge, casting doubt on their suitability for setups of different proximity between source and target domain. Our work makes two contributions. As our first contribution, we re-visit recent FSIG works and their experiments. Our important finding is that, under setups which assumption of close proximity between source and target domains is relaxed, existing state-of-the-art (SOTA) methods which consider only source domain/source task in knowledge preserving perform no better than a baseline fine-tuning method. To address the limitation of existing methods, as our second contribution, we propose Adaptation-Aware kernel Modulation (AdAM) to address general FSIG of different source-target domain proximity. Extensive experimental results show that the proposed method consistently achieves SOTA performance across source/target domains of different proximity, including challenging setups when source and target domains are more apart. Project Page: https://yunqing-me.github.io/AdAM/

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:2210.16559/code)