Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space

Xingwei Qu, Shaowen Wang, Zihao Huang, Kai Hua, Fan Yin, Jundong Zhou, Qiyang Min, Zihao Wang, Yizhi LI, Tianyu Zhang, He Xing, Zheng Zhang, Yuxuan Song, Tianyu Zheng, Zhiyuan Zeng, Chenghua Lin, Ge Zhang, Wenhao Huang

Track: long paper (up to 10 pages)

Keywords: Dynamic Large Concept Models

Abstract: Large language models apply uniform computation to all tokens, despite language exhibiting highly non-uniform information density. We propose Dynamic Large Concept Models (DLCM), which learn variable-length semantic concepts from latent representations and perform reasoning in a compressed concept space. By reallocating computation from redundant token processing to concept-level reasoning, DLCM enables adaptive compute allocation aligned with semantic structure. We further introduce a compression-aware scaling law and a decoupled µP parametrization for heterogeneous token- and concept-level modules. With approximately 34\% lower inference FLOPs, DLCM achieves a 2.69\% average improvement across 12 zero-shot benchmarks, with gains concentrated on reasoning-intensive tasks.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 13