A Simple Sparse Denoising Layer for Robust Deep Learning

Yueming Lyu, Xingrui Yu, Ivor Tsang

Keywords: Sparse Coding, Deep Learning

Abstract: Deep models have achieved great success in many applications. However, vanilla deep models are not well-designed against the input perturbation. In this work, we take an initial step to designing a simple robust layer as a lightweight plug-in for vanilla deep models. To achieve this goal, we first propose a fast sparse coding and dictionary learning algorithm for sparse coding problem with an exact $k$-sparse constraint or $l_0$ norm regularization. Our method comes with a closed-form approximation for the sparse coding phase by taking advantage of a novel structured dictionary. With this handy approximation, we propose a simple sparse denoising layer (SDL) as a lightweight robust plug-in. Extensive experiments on both classification and reinforcement learning tasks manifest the effectiveness of our methods.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: A simple sparse denoising layer as a lightweight plug-in for robust deep learning

Reviewed Version (pdf): https://openreview.net/references/pdf?id=0kLWya6joV