MoLE-GNN: Parameter-Efficient Fine-Tuning of Graph Neural Networks with Mixture-of-Experts

ICLR 2026 Conference Submission18300 Authors

Keywords: Mixture-of-Experts; Parameter-Efficient Fine-Tuning; Transfer Learning; Graph Neural Networks

TL;DR: MoLE-GNN fuses lightweight adapters with GNN experts for depth-adaptive routing, fine-tuning only ~5.1% of params, and outperforms full fine-tuning, PEFT baselines, and prior MoE-GNNs across inductive and link-prediction tasks.

Abstract: Graph Neural Networks (GNNs) are gaining popularity for modeling non-Euclidean data due to their ability to capture local and global structure using message-passing techniques. In real-world scenarios, such as graph classification task, the size of graphs within the same dataset can vary significantly. This warrants an investigation into \emph{depth-sensitivity} of graphs, leading to selection of optimal number of GNN layers according to the size of the graph. Traditional GNNs suffer from a static choice of number of layers for the graphs as it leads to underfitting in the large graphs and overfitting in the smaller ones. Although recent Mixture-of-Experts (MoE) GNN models solve this problem by adaptively selecting depth-sensitive expert networks, they have high computational and memory overhead. To overcome these challenges, we introduce a new hybrid model named MoLE-GNN that combines parameter-efficient adapter modules with GNN experts, supporting dynamic expert assignment with minimal fine-tuning. It drastically minimizes trainable parameters (tunes only 5.1\% of the total parameters) and improves generalization, particularly in low-resource environments. Our extensive experiments across inductive, transductive, and link prediction tasks demonstrate that MoLE-GNN consistently outperforms both full fine-tuning and state-of-the-art PEFT baselines, offering a scalable and effective approach for fine-tuning GNNs on diverse graph topologies. Moreover, MoLE-GNN surpasses existing MoE-based GNNs on inductive and link prediction tasks.

Primary Area: learning on graphs and other geometries & topologies

Submission Number: 18300