A Bregman Proximal Viewpoint on Neural Operators

Abdel-Rahim Mezidi, Jordan Patracone, Saverio Salzo, Amaury Habrard, Massimiliano Pontil, Rémi Emonet, Marc Sebban

Abstract: We present several advances on neural operators by viewing the action of operator layers as the minimizers of Bregman regularized optimization problems over Banach function spaces. The proposed framework allows interpreting the activation operators as Bregman proximity operators from dual to primal space. This novel viewpoint is general enough to recover classical neural operators as well as a new variant, coined Bregman neural operators, which includes the inverse activation operator and features the same expressivity of standard neural operators. Numerical experiments support the added benefits of the Bregman variant of Fourier neural operators for training deeper and more accurate models.

Lay Summary:

Computer programs can struggle when simulating physics phenomena like fluid flow or climate modeling, as it requires learning complex patterns. Recent approaches—called neural operators—demonstrated potential but often hit performance limits when they get bigger, restricting their reliability.

Using ideas from optimization theory, we designed a new perspective on how neural operators work. This viewpoint led us to develop "Bregman Neural Operators", which—unlike conventional approaches—improve as they get larger rather than degrading. Our approach produced more accurate results across a range of challenging physical simulations.

Our work bridges mathematical theory with practical machine learning. This helps scientists run more reliable simulations of the real world and it provides new theoretical insights into how these programs learn, opening pathways for further innovations.

Link To Code: https://github.com/armezidi/bregmano

Primary Area: Optimization->Convex

Keywords: neural operators, proximal optimization, bregman divergence, fourier neural operator

Submission Number: 13247