From Circuits to Dynamics: Understanding and Stabilizing Failure in 3D Diffusion Transformers

ICLR 2026 Conference Submission19744 Authors

Keywords: mechanistic interpretability, diffusion dynamics, test-time intervention, 3D diffusion

Abstract: Reliable surface completion from sparse point clouds underpins many applications spanning content creation and robotics. While 3D diffusion transformers attain state-of-the-art results on this task, we uncover that they exhibit a catastrophic mode of failure: arbitrarily small on-surface perturbations to the input point cloud can fracture the output into multiple disconnected pieces -- a phenomenon we call meltdown. Using activation-patching from mechanistic interpretability, we localize meltdown to a single early denoising cross-attention activation. We find that the singular-value spectrum of this activation provides a scalar proxy: its spectral entropy rises when fragmentation occurs and returns to baseline when patched. Interpreted through diffusion dynamics, we show that this proxy tracks a symmetry-breaking bifurcation of the reverse process. Guided by this insight, we introduce PowerRemap, a drop-in, test-time control that stabilizes sparse point-cloud conditioning. On Google Scanned Objects, PowerRemap has a stabilization rate of 98.3% for the state-of-the-art diffusion transformer WaLa. Overall, this work is a case study on how diffusion model behavior can be understood and guided based on mechanistic analysis, linking a circuit-level cross-attention mechanism to diffusion-dynamics accounts of trajectory bifurcations.

Primary Area: interpretability and explainable AI

Submission Number: 19744