AWDF: An Adaptive Weighted Deep Fusion Architecture for Multi-modality Learning

Qinghan Xue, Abhishek Kolagunda, Steven Eliuk, Xiaolong Wang

Abstract: Fusion has been widely used in machine learning community, especially for problems dealing with multiple input sources and classifiers. The general strategy for information fusion in deep neural network is to directly concatenate the embedding features on the latent space of input sources. However, it is very hard to capture the relative importance of fused sources. It is also impossible to learn the correlation among fused multimodalities inputs, e.g., intra-class and inter-class similarities. Besides, most existing deep learning fusion approaches use universal fusion weights strategy, which cannot fully exploit the relative importance of different inputs. In order to address these problems, in this work we propose an Adaptive Weighted Deep Fusion scheme (AWDF) to capture potential relationships among various input sources. It integrates the feature level and decision level fusion in one framework. Furthermore, in order to address the limitations of existing fusing models with fixed weights, we propose a new scheme named Cross Decision Weights Method (CDWM). It can dynamically learn the weight for each input branch during the fusion process instead of utilizing pre-defined weights. To evaluate the performance of AWDF, we conduct experiments on three different real-world datasets: Wild Business Terms (WBT) Dataset, Iceberg Detection Dataset and CareerCon Dataset. Our experimental results demonstrate the superiority of AWDF over other fusion approaches.