Multi-Task Learning with Hypernetworks and Task Metadata

Olivier Graffeuille, Yun Sing Koh, Jörg Wicker, Moritz K Lehmann

Keywords: multi-task learning, hypernetworks, metadata

TL;DR: A novel multi-task learning architecture which uses a shared hypernetwork to generate entire task-specific network weights from small task embeddings, and can integrate task-level metadata to learn task functions explicitly.

Abstract: Multi-task learning architectures aim to model a set of related tasks simultaneously by sharing parameters across networks to exploit shared knowledge and improve performance. Designing multi-task architectures is challenging due to the trade-off between parameter efficiency and the ability to flexibly model task differences at all network layers. We propose a novel multi-task learning architecture called Multi-Task Hypernetworks, which circumvents this trade-off, generating flexible task networks with a minimal number of parameters per task. Our approach uses a hypernetwork to generate different network weights for each task from task-specific embeddings and enable abstract knowledge transfer between tasks. Our approach stands out from existing multi-task learning architectures by providing the added capability to effectively leverage task-level metadata to explicitly learn task relationships and task functions. We show empirically that Multi-Task Hypernetworks outperform many state-of-the-art multi-task learning architectures on small tabular data problems, and leverage metadata more effectively than existing methods.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1024