DGMamba: Domain Generalization via Generalized State Space Model

Shaocong Long, Qianyu Zhou, Xiangtai Li, Xuequan Lu, Chenhao Ying, Yuan Luo, Lizhuang Ma, Shuicheng YAN

Abstract:

Domain generalization (DG) aims at solving distribution shift problems in various scenes. Existing approaches are based on Convolution Neural Networks (CNNs) or Vision Transformers (ViTs), which suffer from limited receptive fields or quadratic complexity issues.
Mamba, as an emerging state space model (SSM), possesses superior linear complexity and global receptive fields. Despite this, it can hardly be applied to DG to address distribution shifts, due to the hidden state issues and inappropriate scan mechanisms. In this paper, we propose a novel framework for DG, named DGMamba, that excels in strong generalizability toward unseen domains and meanwhile has the advantages of global receptive fields, and efficient linear complexity. Our DGMamba compromises two core components: Hidden State Suppressing (HSS) and Semantic-aware Patch Refining (SPR). In particular, HSS is introduced to mitigate the influence of hidden states associated with domain-specific features during output prediction. SPR strives to encourage the model to concentrate more on objects rather than context, consisting of two designs: Prior-Free Scanning (PFS), and Domain Context Interchange (DCI). Concretely, PFS aims to shuffle the non-semantic patches within images, creating more flexible and effective sequences from images, and DCI is designed to regularize Mamba with the combination of mismatched non-semantic and semantic information by fusing patches among domains. Extensive experiments on four commonly used DG benchmarks demonstrate that the proposed DGMamba achieves remarkably superior results to state-of-the-art models. The code will be made publicly available.

Primary Subject Area: [Generation] Multimedia Foundation Models

Relevance To Conference: Research on domain generalization and efficient multimedia foundation models significantly contributes to multimedia processing by enhancing generalizability of models across diverse media and environments. In multimedia applications, where data often originates from various sources with distinct characteristics such as hand-drawn illustrations, software-composited images, object-centered photographs, and scene-centered shots, domain generalization facilitates more effective analysis, interpretation, and understanding of multimedia content. By tackling the challenge of distributions shifts across diverse media, this work enables models to generalize better to new media. Besides, this work seeks to introduce a novel efficient architecture base on Mamba for domain generalization and multimedia. The objective is to provide multimedia with a more robust backbone that exhibits both superiorities of global receptive fields and linear complexity compared to the prevailing backbones, such as CNNs and ViTs. Overall, advancements in domain generalization and multimedia foundation models play a pivotal role in advancing the capabilities of multimedia processing systems, enhancing their generalization performance in real-world applications.

Submission Number: 415