Improving the Lipschitz stability in Spectral Transformer through Nearest Neighbour Coupling

Anonymous

Keywords: Ising model, Lipschitz stability, Transformer

TL;DR: Inspired from Ising model, nearest neighbour coupling based self-attention is used in transformer to learn RGB to multispectral mapping in stable manner.

Abstract: Ising model has already played a pivotal role in the formulation of neural networks like Hopfield networks and Restricted Boltzmann machines. Recently, Transformers have recently gained popularity due to their ability to learn long range dependencies through self-attention. In our work, we first show that spectral feature learning with self-attention is prone to instability. Inspired from the Ising model, we then propose a transformer based network using a adjacently coupled spectral attention to learn the spectral mapping from RGB images. We further analyse its stability using the theory of Lipschitz constant. The method is evaluated and compared with different state-of-the-art methods on multiple standard datasets.