Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision

Marco Cipriano, Moritz Feuerpfeil, Gerard de Melo

TL;DR: We propose a text-guided SVG generative model that creates vector graphics from natural language descriptions, learning from raster images without direct SVG supervision

Abstract: Scalable Vector Graphics (SVG) is a popular format on the web and in the design industry. However, despite the great strides made in generative modeling, SVG has remained underexplored due to the discrete and complex nature of such data. We introduce GRIMOIRE, a text-guided SVG generative model that is comprised of two modules: A Visual Shape Quantizer (VSQ) learns to map raster images onto a discrete codebook by reconstructing them as vector shapes, and an Auto-Regressive Transformer (ART) models the joint probability distribution over shape tokens, positions and textual descriptions, allowing us to generate vector graphics from natural language. Unlike existing models that require direct supervision from SVG data, GRIMOIRE learns shape image patches using only raster image supervision which opens up vector generative modeling to significantly more data. We demonstrate the effectiveness of our method by fitting GRIMOIRE for closed filled shapes on the MNIST and Emoji, and for outline strokes on icon and font data, surpassing previous image-supervised methods in generative quality and vector-supervised approach in flexibility.

Lay Summary: Grimoire is a new AI model that can create vector drawings like icons or simple illustrations from text descriptions. Vector graphics is the image format used in design and web applications, but teaching computers to make them is challenging because of their complex format. Unlike many existing methods, Grimoire does not need data in vector format to learn from. Instead, we train the model using regular raster images, which are far more common. First, the model learns to reconstruct simple parts of the images as vector data after mapping them into a discrete codebook; then, given a text description, the model learns to predict the correct sequence and position of codes to generate new SVGs. This approach lets Grimoire be very flexible in supporting new SVG attributes while outperforming previous raster supervision methods. We tested our model on handwritten numbers, emojis, and icons, showing it can handle solid shapes and outlines.

Link To Code: https://github.com/potpov/VectorGrimoire

Primary Area: Deep Learning->Generative Models and Autoencoders

Keywords: Image Generation, Scalable Vector Graphics, VQ-VAE, Differentiable Rasterizer

Submission Number: 15874