Sketch2Diagram: Generating Vector Diagrams from Hand-Drawn Sketches

Itsumi Saito, Haruto Yoshida, Keisuke Sakaguchi

Keywords: multimodal, large language model, diagram, vector graphics

TL;DR: We address the challenge of automatically converting hand-drawn sketches into vector diagrams. We present SkeTikZ, a dataset of 3,231 sketch-diagram-code triplets, and ImgTikZ, a vision-language model that performs the sketch-to-diagram conversion.

Abstract:

We address the challenge of automatically generating high-quality vector diagrams from hand-drawn sketches.
Vector diagrams are essential for communicating complex ideas across various fields, offering flexibility and scalability.
While recent research has progressed in generating diagrams from text descriptions, converting hand-drawn sketches into vector diagrams remains largely unexplored due to the lack of suitable datasets.
To address this gap, we introduce SketikZ, a dataset comprising 3,231 pairs of hand-drawn sketches and thier corresponding TikZ codes as well as reference diagrams.
Our evaluations reveal the limitations of state-of-the-art vision and language models (VLMs), positioning SketikZ as a key benchmark for future research in sketch-to-diagram conversion.
Along with SketikZ, we present ImgTikZ, an image-to-TikZ model that integrates a 6.7B parameter code-specialized open-source large language model (LLM) with a pre-trained vision encoder.
Despite its relatively compact size, ImgTikZ performs comparably to GPT-4o.
This success is driven by using our two data augmentation techniques and a multi-candidate inference strategy.
Our findings open promising directions for future research in sketch-to-diagram conversion and broader image-to-code generation tasks. SketikZ is publicly available.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10211