Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks

ICLR 2026 Conference Submission11894 Authors

Keywords: Mixture of Experts, memorization, reasoning, scaling laws, large language models

TL;DR: Memorization skills consistently benefit from higher sparsity, while reasoning skills require balancing active FLOPs with total tokens per parameter; the optimal point shifts with the compute budget.

Abstract:

Empirical scaling laws have driven the evolution of large language models (LLMs), yet their coefficients shift whenever the model architecture or data pipeline changes.
Mixture‑of‑Experts (MoE) models, now standard in state‑of‑the‑art systems, introduce a new sparsity dimension that current dense‑model frontiers overlook.
We investigate how MoE sparsity influences two distinct capability regimes: memorization skills and reasoning skills.
By training MoE families that vary total parameters, active parameters, and top-$k$ routing under fixed compute budgets, we disentangle pre-training loss from downstream accuracy.
Our results reveal two principles. First, Active FLOPs: models with identical training loss but greater active compute achieve higher reasoning accuracy. Second, Total tokens per parameter (TPP): memorization tasks improve with more parameters, while reasoning tasks benefit from optimal TPP, indicating that reasoning is data-hungry.
Neither reinforcement learning post-training (GRPO) nor increased test-time compute alters these trends.
We therefore argue that optimal MoE sparsity must be determined jointly by active FLOPs and TPP, revising the classical picture of compute-optimal scaling.
All code, data sources, and logs are released to facilitate reproducibility and future work.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 11894