AUXILIARY TASK UPDATE DECOMPOSITION: THE GOOD, THE BAD AND THE NEUTRAL

Lucio M. Dery, Yann Dauphin, David Grangier

Keywords: pre-training, multitask learning, deeplearning, gradient decomposition

Abstract:

While deep learning has been very beneficial in data-rich settings, tasks with smaller training set
often resort to pre-training or multitask learning to leverage data from other tasks. In this case,
careful consideration is needed to select tasks and model parameterizations such that updates from
the auxiliary tasks actually help the primary task. We seek to alleviate this burden by formulating a model-agnostic framework that performs fine-grained manipulation of the auxiliary task gradients. We propose to decompose auxiliary updates into directions which help, damage or leave the primary task loss unchanged. This allows weighting the update directions
differently depending on their impact on the problem of interest. We present a novel and efficient algorithm for that
purpose and show its advantage in practice. Our method leverages efficient automatic differentiation
procedures and randomized singular value decomposition for scalability. We show that our framework is
generic and encompasses some prior work as particular cases. Our approach consistently outperforms strong and widely used baselines when leveraging out-of-distribution data for Text and Image classification tasks.

One-sentence Summary: We improve how we use auxiliary task data for model pre-training by decomposing gradient updates into components guided by the primary task

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Code: [ ldery/ATTITTUD](https://github.com/ldery/ATTITTUD)

Data: [CIFAR-10](https://paperswithcode.com/dataset/cifar-10), [CIFAR-100](https://paperswithcode.com/dataset/cifar-100), [CheXpert](https://paperswithcode.com/dataset/chexpert), [IMDb Movie Reviews](https://paperswithcode.com/dataset/imdb-movie-reviews), [ImageNet](https://paperswithcode.com/dataset/imagenet)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2108.11346/code)