Finding Adequate Additional Layer of Auxiliary Task in BERT-Based Multi-task Learning

Takuto Kitamura, Yu Suzuki

Abstract: We find adequate additional layers of auxiliary tasks for BERT-based multi-task learning. Multi-task learning is a method for improving the accuracy of machine learning model by adding auxiliary tasks. Previous studies propose multi-task learning models with auxiliary tasks added to different layers. Our research question is which layer is effective for adding auxiliary tasks to improve accuracy because the answer is still unknown. The aim of this study is to find adequate additional layer of auxiliary tasks that maximizes model accuracy. We use a BERT-base model consisting of twelve layers of Transformer and experiment with seven datasets. Our experimental results show that changing the additional layer of auxiliary tasks improves \(macro\hbox {-}F1\) by up to 5.1% \((p\hbox {-}value=0.019)\). Moreover, our findings suggest that the insertion of auxiliary tasks into layers with the main task’s characteristics increases accuracy.

External IDs:dblp:conf/iiwas/KitamuraS24