Which Tasks Should Be Learned Together in Multi-task Learning?

Trevor Standley, Amir R. Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, Silvio Savarese

TL;DR: We analyze what tasks are best learned together in one network, and which are best to learn separately.

Abstract: Many computer vision applications require solving multiple tasks in real-time. A neural network can be trained to solve multiple tasks simultaneously using 'multi-task learning'. This saves computation at inference time as only a single network needs to be evaluated. Unfortunately, this often leads to inferior overall performance as task objectives compete, which consequently poses the question: which tasks should and should not be learned together in one network when employing multi-task learning? We systematically study task cooperation and competition and propose a framework for assigning tasks to a few neural networks such that cooperating tasks are computed by the same neural network, while competing tasks are computed by different networks. Our framework offers a time-accuracy trade-off and can produce better accuracy using less inference time than not only a single large multi-task neural network but also many single-task networks.

Code: https://anonymous.4open.science/r/6cd16de7-0d82-454f-86ef-b540591cd782/

Keywords: multi-task learning, Computer Vision

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1905.07553/code)