DMORE: Differentiable Mixture-of-Reasoning-Experts with Uncertainty-Guided Multi-Level Routing

Roman Sultimov, Aleksandr Volkov, Mariia Kovalchuk, Yury Maximov

Keywords: Large Language Models, Mixture-of-Experts, Computational Efficiency

TL;DR: DMORE is an LLM architecture that dynamically routes tasks to internal "experts" based on the model's uncertainty

Abstract: Large language models (LLMs) often face inputs of widely varying reasoning difficulty, yet most systems allocate a fixed amount of computation per example. We introduce DMORE (Differentiable Mixture-of-Reasoning-Experts), a unified architecture that dynamically routes inputs to specialized reasoning experts via uncertainty-guided, multi-level gating. Unlike prior mixture-of-experts methods that switch among separate models, DMORE activates one, two, or all experts within a single model based on Monte Carlo dropout–derived uncertainty estimates. A three-stage training procedure first specializes experts on domain-specific reasoning tasks, then calibrates the uncertainty estimator, and finally optimizes the full system end-to-end. Experiments on four reasoning benchmarks show that DMORE matches or surpasses strong chain-of-thought baselines while reducing computation by 28%.

Submission Number: 291