Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision

Moritz Feuerpfeil, Marco Cipriano, Gerard de Melo

Keywords: Image Generation, Scalable Vector Graphics, VQ-VAE, Differentiable Rasterizer

TL;DR: We propose a text-guided SVG generative model that creates vector graphics from natural language descriptions, learning from raster images without direct SVG supervision.

Abstract:

Scalable Vector Graphics (SVG) is a popular format on the web and in the design industry.
However, despite the great strides made in generative modeling,
SVG has remained underexplored due to the discrete and complex nature of such
data. We introduce GRIMOIRE, a text-guided SVG generative model that is comprised
of two modules: A Visual Shape Quantizer (VSQ) learns to map raster
images onto a discrete codebook by reconstructing them as vector shapes, and
an Auto-Regressive Transformer (ART) models the joint probability distribution
over shape tokens, positions and textual descriptions, allowing us to generate vector
graphics from natural language. Unlike existing models that require direct
supervision from SVG data, GRIMOIRE learns shape image patches using only
raster image supervision which opens up vector generative modeling to significantly more data.
We demonstrate the effectiveness of our method by fitting
GRIMOIRE for closed filled shapes on the MNIST and for outline strokes on icon and font data,
surpassing previous image-supervised methods in generative quality and vector-supervised approach in flexibility.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9973