IPT: In-parameter Thinking

Zhou Ziheng, Yuanhao Zeng, Huan Cui, Huacong Tang, Fang Sun, Yizhou Sun, Zilong Zheng, Demetri Terzopoulos, Ying Nian Wu

Keywords: cot, latent, adaptation, reasoning, hypernetwork

TL;DR: a method to perform cot in the latent space

Abstract: We introduce \textbf{In-Parameter Thinking (IPT)}, the first method to perform chain-of-thought reasoning by dynamically adapting model parameters during inference. Given a test instance, IPT generates lightweight parameter adaptations that unlock additional model capacity for each instance. Across mathematical reasoning benchmarks (MATH500, GSM8K etc), IPT delivers substantial performance gains compared to strong baselines. We further run a series of ablation studies to reveal the effects of the key design choices of IPT. IPT opens a new frontier for test-time scaling by reasoning directly in parameter space.

Primary Area: generative models

Code Of Ethics: true

Submission Guidelines: true

Anonymous Url: true

No Acknowledgement Section: true

Submission Number: 4048