Bridging the Data Processing Inequality and Function-Space Variational Inference
Keywords: bayesian deep learning, information theory, function-space variational inference
Blogpost Url: https://iclr-blogposts.github.io/2024/blog/dpi-fsvi/
Abstract: This blog post explores the interplay between the Data Processing Inequality (DPI) and Function-Space Variational Inference (FSVI) within Bayesian deep learning and information theory. After examining the DPI, a cornerstone concept in information theory, and its pivotal role in governing the transformation and flow of information through stochastic processes, we employ its unique connection to FSVI to highlight the FSVI's focus on Bayesian predictive posteriors over parameter space. Throughout the post, theoretical concepts are intertwined with intuitive explanations and mathematical rigor, offering a holistic understanding of these complex topics. The post culminates by synthesizing insights into the significance of predictive priors in model training and regularization, shedding light on their practical implications in areas like continual learning and knowledge distillation. This comprehensive examination not only enriches theoretical understanding but also highlights practical applications in machine learning, making it a valuable read for researchers and practitioners.
Ref Papers: https://openreview.net/forum?id=rkxacs0qY7, https://openreview.net/forum?id=OQs0pLKGGpS, https://proceedings.mlr.press/v162/rudner22a.html
Id Of The Authors Of The Papers: ~Shengyang_Sun2, ~Guodong_Zhang1, ~Jiaxin_Shi1, ~Roger_Baker_Grosse1, ~Tim_G._J._Rudner2, ~Zonghao_Chen1, ~Yee_Whye_Teh, ~Yarin_Gal12
Conflict Of Interest: No connection to the first paper. I was in the same lab as Tim Rudner under Yarin Gal (OATML). It did not contribute to these OATML papers. I left the lab in May 2023 before drafting the text.
Submission Number: 24
Loading