Bidirectional feature enhancement transformer for unsupervised domain adaptation

Zhiwei Hao, Sheng-Sheng Wang, Sifan Long, Yiyang Li, Hao Chai

Abstract: Unsupervised domain adaptation (UDA) aims to generalize knowledge learned from one labeled source domain to another unlabeled target domain. To extract domain-invariant feature representations, most existing UDA approaches leverage convolution neural network-based frameworks either at the domain level or the category level. Compared with coarse-grained domain-level alignment methods, fine-grained category-level UDA approaches facilitates more precise alignment. However, a basic difficulty with category-level-based UDA is that the construction of pseudo-labels for unlabeled target domain is frequently too noisy for appropriate domain alignment, hence compromising the UDA performance. Clustering methods can be used to extract prototype representations across and within domains, which can be used to augment features in source and target domains. As a result, correlative features can be constructed in the feature space and more robust pseudo-labels can be obtained, allowing for more precise feature alignment. With the significant application of the transformer in visual tasks, the cross-attention mechanism of the transformer is more resistant to noise, making it possible to achieve more accurate feature alignment. In our paper, we propose a bidirectional feature enhancement transformer (BFET) as a solution to the difficult UDA tasks. Specifically, we propose a bidirectional cross-attention transformer architecture with an equilibrium factor to achieve feature representation learning and domain alignment. And we design a bidirectional cross-domain homogeneous prototype feature enhancement algorithm to produce proper Source-Target pairs. Besides, we propose a Feature Enhancement Module to utilize the knowledge of prototypes to provide more accurate pseudo-labels. The proposed BFET can mandate the framework to concurrently learn discriminative representations that are domain-specific as well as feature representations that are domain-invariant. In addition, detailed and systematic experiments illustrate that our BFET achieves noteworthy performance.