UNDERSTANDING PURE CLIP GUIDANCE FOR VOXEL GRID NERF MODELS

Han-Hung Lee, Angel X Chang

Keywords: Text to 3D Generation, CLIP, NeRF, Adversarial Examples, Augmentation

TL;DR: We explore various mechanics that prevent adversarial generations from using CLIP as guidance for training a voxel grid NeRF model without any datasets.

Abstract: We explore the task of text to 3D object generation using CLIP. Specifically, we use CLIP for guidance without access to any datasets, a setting we refer to as pure CLIP guidance. While prior work has adopted this setting, there is no systematic study of mechanics for preventing adversarial generations within CLIP. We illustrate how different image-based augmentations prevent the adversarial generation problem, and how the generated results are impacted. We test different CLIP model architectures and show that ensembling different models for guidance can prevent adversarial generations within bigger models and generate sharper results. Furthermore, we implement an implicit voxel grid model to show how neural networks provide an additional layer of regularization, resulting in better geometrical structure and coherency of generated objects. Compared to prior work, we achieve more coherent results with higher memory efficiency and faster training speeds.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Generative models

Community Implementations: [ 5 code implementations](https://www.catalyzex.com/paper/understanding-pure-clip-guidance-for-voxel/code)