Modal-balanced training strategy by gradient adjustment and artificial Gauss noise enhancement
- Keywords: audio-visual learning, gradient adjustment, imbalanced training, Gauss noise
- Abstract: Audio-visual learning helps to hold a comprehensive understanding of the world, utilizing richer information. However, the strong modality perhaps suppresses the potential of the weak one in the audio-visual learning process, causing its inadequate training and unsatisfactory audio-visual performance. In this paper, we propose a novel Balanced-Training Strategy to improve the training of multiple modalities in a more balanced way. The gradient in the backward is adjusted according to a ratio that measures imbalanced degree between different modalities, aiming to balance the training process. Furthermore, an artificial Gauss noise is introduced in the optimization to enhanced the generalization ability of the model. Experiments on multiple datasets achieve considerable improvement and illustrate the effectiveness of our strategy. Ablation studies are provided to prove the module effectiveness of our strategy.
- One-sentence Summary: We propose a Balanced-Training Strategy which adjusts the gradient of different modalities to achieve more balanced training result.