Periodic Activation Functions Induce StationarityDownload PDF

21 May 2021, 20:50 (edited 20 Dec 2021)NeurIPS 2021 PosterReaders: Everyone
  • Keywords: Bayesian deep learning, Gaussian process, uncertainty quantification
  • TL;DR: Periodic activation functions induce a connection between the prior on the network weights and stationary Gaussian process priors.
  • Abstract: Neural network models are known to reinforce hidden data biases, making them unreliable and difficult to interpret. We seek to build models that `know what they do not know' by introducing inductive biases in the function space. We show that periodic activation functions in Bayesian neural networks establish a connection between the prior on the network weights and translation-invariant, stationary Gaussian process priors. Furthermore, we show that this link goes beyond sinusoidal (Fourier) activations by also covering triangular wave and periodic ReLU activation functions. In a series of experiments, we show that periodic activation functions obtain comparable performance for in-domain data and capture sensitivity to perturbed inputs in deep neural networks for out-of-domain detection.
  • Supplementary Material: pdf
  • Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
  • Code: https://github.com/AaltoML/PeriodicBNN
10 Replies

Loading