Independent Component Alignment for Multi-task Learning

Dmitry Senushkin, Iaroslav Melekhov, Mikhail Romanov, Anton Konushin, Juho Kannala, Arno Solin

Keywords: multi-task learning, optimization

Abstract: We present a novel gradient-based multi-task learning (MTL) approach that balances training in multi-task systems by aligning the independent components of the training objective. In contrast to state-of-the-art MTL approaches, our method is stable and preserves the ratio of highly correlated tasks gradients. The method is scalable, reduces overfitting, and can seamlessly handle multi-task objectives with a large difference in gradient magnitudes. We demonstrate the effectiveness of the proposed approach on a variety of MTL problems including digit classification, multi-label image classification, camera relocalization, and scene understanding. Our approach performs favourably compared to other gradient-based adaptive balancing methods, and its performance is backed up by theoretical analysis.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2305.19000/code)