% This study shows the limitations of BNNs in capturing flatness, which is crucial for generalization ability. We also show that BMA can fail to yield optimal results without explicitly considering flatness. To address this issue, we introduce Flat Posterior-aware Bayesian Model Averaging (FP-BMA), which seeks to find a flat posterior by capturing flatness in the parameter space. FP-BMA is the generalized version of existing sharpness-aware optimizers for DNNs and aligns with the intrinsic nature of BNNs. We further propose a Flat Posterior-aware Bayesian Transfer Learning scheme, which effectively enhances robustness against model misspecification, combined with FP-BMA. 

% While our theoretical analysis provides useful insights, it is subject to certain limitations due to strong assumptions. Empirically, we demonstrate through extensive experiments that FP-BMA significantly enhances the generalization ability of BNNs, though our evaluation does not cover the full spectrum of MCMC algorithms. Overall, our work emphasizes the critical role of flatness in posterior approximations, shedding light on its impact on generalization ability. Thus, we propose a remedy that leverages this insight to enhance both the predictive robustness and accuracy of BNNs.


This study demonstrates the limitations of BNNs in capturing flatness—a property crucial for generalization—and reveals that BMA may fail to yield optimal results without considering flatness. To address this, we introduce FP-BMA, which seeks a flat posterior by effectively capturing flatness in the parameter space. FP-BMA generalizes existing sharpness-aware optimizers and aligns with the intrinsic nature of BNNs. We further propose a Flat Posterior-aware Bayesian Transfer Learning scheme, which enhances resilience against model misspecification. Our extensive experiments demonstrate that FP-BMA significantly improves the generalization ability of BNNs, underscoring the importance of flatness in posterior approximations. However, there are several limitations to our study. Specifically, our theoretical insights rely on strong assumptions, and the empirical evaluation does not cover the full spectrum of MCMC algorithms. Future work could extend FP-BMA to a wider variety of Bayesian inference methods and investigate its
effectiveness on more complex datasets. Additionally, exploring automated ways to quantify and enforce flatness during model training could further enhance the robustness and applicability of the proposed approach.