StyleCL : Latent Dictionary Learning for StyleGAN Without Forgetting

Adarsh Kappiyath, ANMOL GARG, Ramya Hebbalaguppe, Prathosh AP

Keywords: Continual Learning, Generative Modelling

TL;DR: A new method for enabling continual generation from a stream of datasets without forgetting.

Abstract: StyleGAN is one of the most versatile generative models that have emerged in recent times. However, when it is trained continually on a stream of data (potentially previously unseen distributions), it tends to forget the distribution it has learned, as is the case with any other generative model, due to catastrophic forgetting. Recent studies have shown that the latent space of StyleGAN is very versatile, as data from a variety of distributions can be inverted onto it. In this paper, we propose to leverage this property to facilitate lifelong learning of StyleGAN without forgetting. Specifically, given a StyleGAN trained on a certain task (dataset), we propose to learn a set of dictionary vectors in its latent space, one for each novel, unseen task (or dataset). Additionally, we also learn a relatively small set of shared parameters (feature adaptors) in the weight space to complement the dictionary learning in the latent space. During inference, given a dataset/task, our method invokes the corresponding learned latent dictionary and the shared parameters for that particular task. Our method avoids catastrophic forgetting because the set of dictionary and the feature adaptor parameters are unique for each task. However, the generator for each task shares all of the parameters except for the newly added parameters of the feature adaptor. We demonstrate that our method, StyleCL, achieves better generation quality on multiple datasets. Additionally, our method requires significantly fewer additional parameters per task compared to previous methods. This is a consequence of learning task-specific dictionaries in the latent space, which has a much lower dimensionality compared to the weight space. We also demonstrate that our method, StyleCL, offers the capability for positive forward transfer when the tasks are semantically similar.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1374