Generative Blocks World: Moving Things Around in Pictures

Vaibhav Vavilala, Seemandhar Jain, Rahul Vasanth, David Forsyth, Anand Bhattad

Keywords: 3D primitives, Controlled Image Synthesis, Diffusion Models

TL;DR: We can use 3D primitive abstractions to control diffusion models

Abstract: We describe Generative Blocks World to interact with the scene of a generated image by manipulating simple geometric abstractions. Our method represents scenes as assemblies of convex 3D primitives, and the same scene can be represented by different numbers of primitives, allowing an editor to move either whole structures or small details. Once the scene geometry has been edited, the image is generated by a flow-based method which is conditioned on depth and a texture hint. Our texture hint takes into account the modified 3D primitives, exceeding texture-consistency provided by existing techniques. These texture hints (a) allow accurate object and camera moves and (b) preserve the identity of objects. Our experiments demonstrate that our approach outperforms prior works in visual fidelity, editability, and compositional generalization.

Submission Number: 20