Detecting Geometric Deformation in Visual Generation

Leslie Gu, Junhwa Hur, Charles Herrmann, Fangneng Zhan, Todd Zickler, Deqing Sun, Hanspeter Pfister

Keywords: geometric consistency, multi-view consistency, image generation, video generation, optical flow, 3D reconstruction

Abstract: Recent text-to-video and multi-view generative models produce striking imagery but often violate basic 3D geometry, exhibiting non-rigid ``melting'' or ``breathing'' artifacts across viewpoints. We study this failure mode in the static-scene regime, where camera motion is allowed but objects must remain rigid; any apparent object motion is deemed deformation. We introduce a geometry-grounded detection pipeline that localizes and quantifies such artifacts. The pipeline estimates camera motion and depth to predict the rigid pixel motion expected in a static world, compares it to observed optical flow to obtain a motion error map, and fuses this with a depth reprojection error map to handle occlusions. The result is an occlusion-aware, per-pixel deformation map and interpretable video-level scores. To enable controlled, quantitative evaluation, we present \textsc{WarpBench}, a synthetic dataset that applies localized thin-plate-spline warps to real frames while recording dense displacement ground truth. We instantiate it as \textit{CO3D-Warp} (object-centric) and \textit{ScanNet++-Warp} (scene-level). To probe performance beyond synthetic perturbations, we further introduce \textbf{Geo-Flaw}, a task-oriented benchmark spanning object-centric reconstruction, indoor navigation, large-scale outdoor scenes, and challenging surfaces, under both slow and fast camera motion. Our experiments show that the proposed pipeline detects deformation artifacts missed by feature-based metrics and coordinate-only consistency measures, and it naturally extends to moving object segmentation, outperforming prior training-free baselines. Together, these components provide an interpretable and practical toolkit for diagnosing geometric inconsistency and for benchmarking video generative models on true 3D fidelity.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 7863