Robust Core-periphery Constrained Transformer for Domain Adaptation

Xiaowei Yu, Lu Zhang, Yao Xue, Yanjun Lyu, Tianming Liu, Dajiang Zhu

Keywords: Transformer, Core-peiphery, Noisy Brain

Abstract: Unsupervised domain adaptation (UDA) aims to learn transferable representation across domains. Recently a few UDA works have successfully applied Transformer-based methods and achieved state-of-the-art (SOTA) results. However, it remains challenging when there exists a large domain gap between the source and target domain. Inspired by the remarkable transferability abilities of humans, where knowledge can adapt from familiar to uncharted domains, we endeavor to apply universally existing brain structure and function principles, specifically, the core-periphery principle and the concept of the noisy brain, to design and enhance the Transformer, ultimately improving its performance in UDA. In this work, we propose a novel brain-inspired robust core-periphery constrained transformer (RCCT) for unsupervised domain adaptation, which brings a large margin of performance improvement on various datasets. The application of the core-periphery principle and the development of the latent feature interaction (LFI) operation correspond to the `Core-periphery' and `Robust' aspects mentioned in the title. Specifically, in RCCT, the self-attention operation across image patches is rescheduled by an adaptively learned weighted graph with the Core-Periphery structure (CP graph), where the information communication and exchange between image patches are manipulated and controlled by the connection strength, i.e., edge weight of the learned weighted CP graph. In addition, considering the noisy nature of data in domain adaptation tasks, we propose a latent feature interaction operation to enhance model robustness, wherein we intentionally introduce perturbations to the latent features in the latent space, ensuring the generation of robust learned weighted core-periphery graphs. We conducted extensive evaluations on several well-established UDA benchmarks, and the experimental results demonstrate that applying brain-inspired principles leads to promising results, surpassing the performance of existing Transformer-based methods.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4100