Exploring the Sharpened Cosine Similarity

Skyler Wu, Fred Lu, Edward Raff, James Holt

Keywords: deep-learning, sharpened-cosine-similarity, interpretability, adversarial-robustness

TL;DR: While Sharpened Cosine Similarity may not be more accurate than standard convolution layers, it may learn more interpretable representations and be more adversarially robust under certain circumstances.

Abstract: Convolutional layers have long served as the primary workhorse for image classification. Recently, an alternative to convolution was proposed using the Sharpened Cosine Similarity (SCS), which in theory may serve as a better feature detector. While multiple sources report promising results, there has not been to date a full-scale empirical analysis of neural network performance using these new layers. In our work, we explore SCS's parameter behavior and potential as a drop-in replacement for convolutions in multiple CNN architectures benchmarked on CIFAR-10. We find that while SCS may not yield significant increases in accuracy, it may learn more interpretable representations. We also find that, in some circumstances, SCS may confer a slight increase in adversarial robustness.

Community Implementations: [ 7 code implementations](https://www.catalyzex.com/paper/exploring-the-sharpened-cosine-similarity/code)