TENSOR RING NETS ADAPTED DEEP MULTI-TASK LEARNING

Xinqi Chen, Ming Hou, Guoxu Zhou, Qibin Zhao

Abstract: Recent deep multi-task learning (MTL) has been witnessed its success in alleviating data scarcity of some task by utilizing domain-specific knowledge from related tasks. Nonetheless, several major issues of deep MTL, including the effectiveness of sharing mechanisms, the efficiency of model complexity and the flexibility of network architectures, still remain largely unaddressed. To this end, we propose a novel generalized latent-subspace based knowledge sharing mechanism for linking task-specific models, namely tensor ring multi-task learning (TRMTL). TRMTL has a highly compact representation, and it is very effective in transferring task-invariant knowledge while being super flexible in learning task-specific features, successfully mitigating the dilemma of both negative-transfer in lower layers and under-transfer in higher layers. Under our TRMTL, it is feasible for each task to have heterogenous input data dimensionality or distinct feature sizes at different hidden layers. Experiments on a variety of datasets demonstrate our model is capable of significantly improving each single task’s performance, particularly favourable in scenarios where some of the tasks have insufficient data.

Keywords: deep learning, deep multi-task learning, tensor factorization, tensor ring nets

TL;DR: a deep multi-task learning model adapting tensor ring representation

Data: [CIFAR-10](https://paperswithcode.com/dataset/cifar-10)