Multi-Task Learning by Deep Collaboration and Application in Facial Landmark Detection

Ludovic Trottier, Philippe Giguère, Brahim Chaib-draa

Abstract: Convolutional neural networks (CNN) have become the most successful and popular approach in many vision-related domains. While CNNs are particularly well-suited for capturing a proper hierarchy of concepts from real-world images, they are limited to domains where data is abundant. Recent attempts have looked into mitigating this data scarcity problem by casting their original single-task problem into a new multi-task learning (MTL) problem. The main goal of this inductive transfer mechanism is to leverage domain-specific information from related tasks, in order to improve generalization on the main task. While recent results in the deep learning (DL) community have shown the promising potential of training task-specific CNNs in a soft parameter sharing framework, integrating the recent DL advances for improving knowledge sharing is still an open problem. In this paper, we propose the Deep Collaboration Network (DCNet), a novel approach for connecting task-specific CNNs in a MTL framework. We define connectivity in terms of two distinct non-linear transformation blocks. One aggregates task-specific features into global features, while the other merges back the global features with each task-specific network. Based on the observation that task relevance depends on depth, our transformation blocks use skip connections as suggested by residual network approaches, to more easily deactivate unrelated task-dependent features. To validate our approach, we employed facial landmark detection (FLD) datasets as they are readily amenable to MTL, given the number of tasks they include. Experimental results show that we can achieve up to 24.31% relative improvement in landmark failure rate over other state-of-the-art MTL approaches. We finally perform an ablation study showing that our approach effectively allows knowledge sharing, by leveraging domain-specific features at particular depths from tasks that we know are related.

TL;DR: We propose a novel approach for connecting task-specific networks in a multi-task learning setting based on recent residual network advances.

Keywords: multi-task learning, soft parameter sharing, facial landmark detection

Code: [ ltrottier/deep-collaboration-network](https://github.com/ltrottier/deep-collaboration-network)

Data: [AFLW](https://paperswithcode.com/dataset/aflw), [AFW](https://paperswithcode.com/dataset/afw)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/multi-task-learning-by-deep-collaboration-and/code)