Revisiting Hilbert-Schmidt Information Bottleneck for Adversarial Robustness

Zifeng Wang, Tong Jian, Aria Masoomi, Stratis Ioannidis, Jennifer Dy

Keywords: Adversarial Robustness, Hilbert-Schmidt Independence Criterion, Information Bottleneck, Deep Learning

Abstract: We investigate the HSIC (Hilbert-Schmidt independence criterion) bottleneck as a regularizer for learning an adversarially robust deep neural network classifier. In addition to the usual cross-entropy loss, we add regularization terms for every intermediate layer to ensure that the latent representations retain useful information for output prediction while reducing redundant information. We show that the HSIC bottleneck enhances robustness to adversarial attacks both theoretically and experimentally. In particular, we prove that the HSIC bottleneck regularizer reduces the sensitivity of the classifier to adversarial examples. Our experiments on multiple benchmark datasets and architectures demonstrate that incorporating an HSIC bottleneck regularizer attains competitive natural accuracy and improves adversarial robustness, both with and without adversarial examples during training. Our code and adversarially robust models are publicly available.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

TL;DR: We show that a penalty based on the Hilbert-Schmidt Independence Criterion enhances adversarial robustness both theoretically and experimentally.

Code: https://github.com/neu-spiral/HBaR