SketchEdit: Editing Freehand Sketches At The Stroke-Level

Tengjie Li, Shikui Tu, Lei Xu

Keywords: Sketch synthesis，sketch edit， sketch representation learning， diffusion model

Abstract: Freehand sketching is a representation of human cognition of the real world. Recent sketch synthesis methods have demonstrated the capability of generating lifelike outcomes. However, these methods directly encode the whole sketch instances and makes it challenging to decouple the strokes from the sketches and have difficulty in controlling local sketch synthesis, e.g., stroke editing. Besides, the sketch editing task encounters the issue of accurately positioning the edited strokes, because users may not be able to draw on the exact position and the same stroke may appear on various locations in different sketches. We propose SketchEdit to realize flexible editing of sketches at the stroke-level for the first time. To tackle the challenge of decoupling strokes, our SketchEdit divides a drawing sequence of a sketch into a series of strokes based on the pen state, align the stroke segments to have the same starting position, and learns the embeddings of every stroke by a proposed stroke encoder. This design allows users to conveniently select the strokes for editing at any locations. Moreover, we overcome the problem of stroke placement via a diffusion process, which progressively generate the locations for the strokes to be synthesized, using the stroke features as the guiding condition. Both the stroke embeddings and the generated locations are fed into a sequence decoder to synthesize the manipulated sketch. The stroke encoder and the sequence decoder are jointly pre-trained under the autoencoder paradigm, with an extra image decoder to learn the local structure of sketches. Experiments demonstrate that the SketchEdit is effective for stroke-level sketch editing and outperforms state-of-the-art methods in the sketch reconstruction task.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7083