Frustratingly easy quasi-multitask learning

Gábor Berend, Norbert Kis-Szabó

TL;DR: We propose a computationally efficient alternative for traditional ensemble learning for neural nets.

Abstract: We propose the technique of quasi-multitask learning (Q-MTL), a simple and easy to implement modification of standard multitask learning, in which the tasks to be modeled are identical. We illustrate it through a series of sequence labeling experiments over a diverse set of languages, that applying Q-MTL consistently increases the generalization ability of the applied models. The proposed architecture can be regarded as a new regularization technique encouraging the model to develop an internal representation of the problem at hand that is beneficial to multiple output units of the classifier at the same time. This property hampers the convergence to such internal representations which are highly specific and tailored for a classifier with a particular set of parameters. Our experiments corroborate that by relying on the proposed algorithm, we can approximate the quality of an ensemble of classifiers at a fraction of computational resources required. Additionally, our results suggest that Q-MTL handles the presence of noisy training labels better than ensembles.

Code: https://drive.google.com/drive/folders/16ORV5A0Zqo52h0vXt2eJwyGPZBmtRNWg?usp=sharing

Keywords: multitask learning, ensembling