SFP: Spurious Feature-Targeted Pruning for Out-of-Distribution Generalization

Yingchun Wang, Jingcai Guo, Song Guo, LIU Yi, Jie ZHANG, Weizhan Zhang

Abstract: Recent studies reveal that even highly biased dense networks can contain an invariant substructure with superior out-of-distribution (OOD) generalization. While existing works commonly seek these substructures using global sparsity constraints, the uniform imposition of sparse penalties across samples with diverse levels of spurious contents renders such methods suboptimal. The precise adaptation of model sparsity, specifically tailored for spurious features, remains a significant challenge. Motivated by the insight that in-distribution (ID) data containing spurious features may exhibit lower experiential risk, we propose a novel **S**purious **F**eature-targeted **P**runing framework, dubbed **SFP**, to induce the authentic invariant substructures without referring to the above concerns. Specifically, SFP distinguishes spurious features within ID instances during training by a theoretically validated threshold. It then penalizes the corresponding feature projections onto the model space, steering the optimization towards subspaces spanned by those invariant factors. Moreover, we also conduct detailed theoretical analysis to provide a rationality guarantee and a proof framework for OOD structures based on model sparsity. Experiments on various OOD datasets show that SFP can significantly outperform both structure-based and non-structure-based OOD generalization SOTAs by large margins.

Relevance To Conference: First, out-of-distribution (OOD) is a critical challenge in multimedia or multimodal learning, where data across training and testing domains can vary frequently and dynamically. OOD generalization can significantly mitigate this issue and contribute to a more stable and generalized model. Second, network pruning or substructure learning is also helpful in multimedia/multimodal processing on low-resource devices in terms of computing capability, memory, and data, which helps to fulfill the real-time requirement with all real-world applications. Last but not the least, this work also provides a solid proof framework for OOD structures based on model sparsity and is also verified on various datasets.

Primary Subject Area: [Content] Vision and Language

Secondary Subject Area: [Experience] Multimedia Applications

Submission Number: 2007