CLIPDraw++: Text-to-Sketch Synthesis with Simple Primitives

Nityanand Mathur, Shyam Marjit, Abhra Chaudhuri, Anjan Dutta

Keywords: Concept Visualization, Sketch Synthesis, Geometric Primitives, Linear Transformations, CLIP Embeddings, Vector Graphics, Optimization-based Drawing, Explainable AI

TL;DR: CLIPDraw++ visualizes CLIP's latent space through simple primitives, enabling clear and interpretable sketch synthesis.

Abstract: With the goal of understanding the visual concepts that CLIP associates with text prompts, we show that the latent space of CLIP can be visualized solely in terms of linear transformations on simple geometric primitives like straight lines and circles. Although existing approaches achieve this by sketch-synthesis-through-optimization, they do so on the space of Bézier curves, which exhibit a wastefully large set of structures that they can evolve into, as most of them are non-essential for generating meaningful sketches. We present CLIPDraw++, an algorithm that provides significantly better visualizations for CLIP text embeddings, using only simple primitive shapes like straight lines and circles. This constrains the set of possible outputs to linear transformations on these primitives, thereby exhibiting an inherently simpler mathematical form. The synthesis process of CLIPDraw++ can be tracked end-to-end, with each visual concept being expressed exclusively in terms of primitives. Implementation will be released upon acceptance.

Submission Number: 16