Modality-Specific Learning Rate Control for Multimodal Classification

Naotsuna Fujimori, Rei Endo, Yoshihiko Kawai, Takahiro Mochizuki

Abstract: Multimodal machine learning is an approach to performing tasks with inputs containing multiple expressions for a single subject. There are many recent reports on multimodal machine learning using the framework of deep neural networks. Conventionally, a common learning rate has been used for the network of all modalities. This has led, however, to a decrease in the overall accuracy due to overfitting in some modality models in cases when the convergence rate and generalization performance differ among modalities. In this paper, we propose a method that solves this problem by constructing a model within the framework of multitask learning, which simultaneously learns modality-specific classifiers as well as a multimodal classifier, to detect overfitting in each modality and carry out early stopping separately. We evaluated the accuracy of the proposed method using several datasets and demonstrated that it improves classification accuracy.