Nestor: A Nested MOE-based Neural Operator for Large-Scale PDE Pre-Training

Dengdi Sun, Xiaoya Zhou, Xiao Wang, Hao Si, wanli lyu, Jin Tang

Keywords: Pre-trained Neural Operators, Mixture-of-Experts (MoE), PDE Modeling, Operator Learning

Abstract: Neural operators have emerged as an efficient paradigm for solving PDEs, overcoming the limitations of traditional numerical methods and significantly improving computational efficiency. However, due to the diversity and complexity of PDE systems, existing neural operators typically rely on a single network architecture, which limits their capacity to fully capture heterogeneous features and complex system dependencies. This constraint poses a bottleneck for large-scale PDE pre-training based on neural operators. To address these challenges, we propose a large-scale PDE pre-trained neural operator based on a nested Mixture-of-Experts (MoE) framework. In particular, the image-level MoE is designed to capture global dependencies, while the token-level Sub-MoE focuses on local dependencies. Our model can selectively activate the most suitable expert networks for a given input, thereby enhancing generalization and transferability. We conduct large-scale pre-training on twelve PDE datasets from diverse sources and successfully transfer the model to downstream tasks. Extensive experiments demonstrate the effectiveness of our approach.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 10418