MUX: Continuous Reasoning via Multiplexed Tokens

Ayhan Suleymanzade, Halil Alperen Gozeten, Ismail Ilkan Ceylan, Jinwoo Kim

Track: long paper (up to 10 pages)

Keywords: Continuous Reasoning, Efficient Reasoning, Chain-of-Thought (CoT), Large Language Models (LLMs)

Abstract: Language models solve complex problems by articulating intermediate reasoning steps in natural language. While effective, this process is computationally bottlenecked: each reasoning step conveys only a single subword, and many steps are spent expressing a thought rather than carrying out computation. We propose MUX, a simple method for high-bandwidth and compact reasoning based on distillation of discrete reasoning into continuous multiplexed tokens. Each continuous token is trained to represent a weighted linear superposition (multiplexing) of a span of discrete reasoning steps, while ensuring that this superposition is lossless and the span can be fully recovered (demultiplexing). We prove that simple position-dependent weightings, such as properly chosen geometric decay, support lossless multiplexing, and further prove that multiplexed reasoning can perform parallel exploration in problems that require search. Across 16 evaluation settings spanning two language models, MUX is competitive with or outperforms strong continuous reasoning baselines, especially when the base discrete reasoning is verbose. Overall, our results suggest that appropriately chosen learning targets can make continuous reasoning both efficient and interpretable.

Presenter: ~Ayhan_Suleymanzade1

Format: Yes, the presenting author will definitely attend in person because they attending ICLR for other complementary reasons.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 206