MinMax Bayesian Neural Networks and Uncorrelated Representation

Junping Hong, Yutong Jiang, Ercan Engin KURUOGLU

Keywords: Minimax game, Bayesian Neural Networks, Brownian Motion, Minimax coding reduction, Uncorrelated representation

TL;DR: Apply minimax method with the minimax coding rate loss to Bayesian neural network to verify previous stament in previous paper and use sampling to visualize the bias-variance tradeoff.

Abstract: In deep learning, Bayesian neural networks (BNN) and dropout techniques provide the role of robustness analysis, and the minimax method used to be a conservative choice in the traditional Bayesian field. In this paper, we apply the minimax game to the BNN on the representation level and formulate as a two-player game between a deterministic neural network $f$ and a sampling stochastic neural network $f + r*\xi$, which can be seen as a Brownian Motion of $f$. Our simple experiments show that $r$ will be stable with enough dimension space, suitable activation function, and without bias with the minimax coding rate loss, which verify the statement \cite{yu2020learning} in some sense. And we test the convolutional neural network without bias, with bias and with batch normalization on simple data set like MNIST, Fashion MNIST and others, and visualize the sampling radius as a bias-variance tradeoff study. At last, we also test how noise perturbation will affect radius in stable case.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5490