3D Shape Tokenization via Latent Flow Matching

Jen-Hao Rick Chang, Yuyang Wang, Miguel Ángel Bautista, Jiatao Gu, Xiaoming Zhao, Joshua M. Susskind, Oncel Tuzel

Abstract:

We introduce a latent 3D representation that models 3D surfaces as probability density functions in 3D, i.e., $p(x, y, z)$, with flow-matching. Our representation is specifically designed for consumption by machine learning models, offering continuity and compactness by construction while requiring only point clouds and minimal data preprocessing. Despite being a data-driven method, our use
of flow matching in the 3D space enables interesting geometry properties, including the capabilities to perform zero-shot estimation of surface normal and deformation field. We evaluate with several machine learning tasks, including 3DCLIP, unconditional generative models, single-image conditioned generative model, and intersection-point estimation. Across all experiments, our models achieve competitive performance to existing baselines, while requiring less preprocessing and auxiliary information from training data.