LASER: Low-Rank Activation SVD for Efficient Recursion

Ege Çakar, Ketan Raghu, Lia Zheng

Track: long paper (up to 10 pages)

Keywords: implicit reasoning, recursive models, looped architectures, activation compression, low-rank approximation, subspace tracking, memory efficiency

TL;DR: TRM activations during recursive reasoning occupy a low-dimensional linear subspace; we exploit this via adaptive subspace tracking to achieve 60% memory savings with no accuracy loss.

Abstract:

Recursive architectures such as Tiny Recursive Models (TRMs)
perform implicit reasoning through iterative latent computation,
yet the geometric structure of these reasoning trajectories
remains poorly understood. We investigate the activation manifold
of TRMs during recursive unrolling and find that activations
occupy an effectively linear, low-dimensional subspace whose
principal directions can be tracked dynamically with cheap power
iterations. This suggests that weight-sharing concentrates
iterative computation along a small number of dominant
eigendirections, and we find that this concentration varies
sharply across computational sites. We exploit this structure
through LASER (Low-Rank Activation SVD for Efficient Recursion),
a dynamic compression framework that maintains an evolving
low-rank basis via matrix-free subspace tracking with a
fidelity-triggered reset mechanism, achieving ${\sim}60\%$
activation memory savings with no statistically significant
accuracy degradation. Our analysis raises questions about how
recursive architectures allocate representational capacity during
implicit reasoning, and whether this concentration can be
exploited to improve the efficiency and stability of latent
computation.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Presenter: ~Ege_Çakar1

Format: Maybe: the presenting author will attend in person, contingent on other factors that still need to be determined (e.g., visa, funding).

Funding: Yes, the presenting author of this submission falls under ICLR’s funding aims, and funding would significantly impact their ability to attend the workshop in person.

Submission Number: 56