Towards Adversarially Robust CLIP: A Hierarchical Model Fusion Method Using Optimal Transport

ICLR 2026 Conference Submission15780 Authors

Keywords: adversarial robustness, optimal transport, model fusion

TL;DR: We improve CLIP’s adversarial robustness by hierarchically fusing diverse, attack- and prompt-specific submodels using optimal transport.

Abstract: In recent years, multimodal models such as CLIP have achieved impressive performance but remain vulnerable to adversarial perturbations. Although adversarial training can enhance robustness, it often leads to overfitting toward specific attack types. One solution for improving generalization is to integrate multiple diverse and adversarially trained submodels, but this strategy could incur high test-time cost. To achieve a promising tradeoff between robust generalization and efficiency, we consider to design an optimal transport (OT) based model fusion method, which is called ``HOT-CLIP (Hierarchical Optimal Transport Fusion for CLIP)". Although several OT based model fusion methods have been proposed before, they cannot be easily adapted to solve our problem, since they may suffer the issues like parameter misalignment when dealing with highly diverse and multimodal submodels. Our proposed method constructs diverse submodels by varying both attack methods and textual prompts, and integrates them via a hierarchical two-level OT fusion method. The intra-attack fusion first aligns and merges models within the same attack family, and the inter-attack fusion subsequently combines these aligned models across different attacks. Through this carefully crafted fusion strategy, HOT-CLIP can significantly improve the accuracy for alignment and reduce the total occupied memory. More importantly, the obtained robust visual encoder can be deployed without additional inference-time cost. In our experiments, the results on multiple vision-language tasks demonstrate that HOT-CLIP can greatly enhance the model's adversarial robustness while maintaining competitive clean accuracy.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 15780