ProTransformer: Robustify Transformers via Plug-and-Play Paradigm

Zhichao Hou, Weizhi Gao, Yuchen Shen, Xiaorui Liu

Keywords: Adversarial Robustness, Transformers, Large Language Models

Abstract: Transformer-based architectures have dominated many machine learning areas in recent years. In this paper, we propose a simple yet highly effective robust attention mechanism to robustify any transformer-based architectures. Our algorithm can be implemented with only 4 lines of code and be plugged into any given transformer as a plug-and-play layer to enhance its robustness without additional training or fine-tuning. Comprehensive experiments and ablation studies show that the proposed ProTransformer significantly improves the robustness across various prediction tasks, attack mechanisms, backbone architectures, and data domains.

Submission Number: 39