Learning assisted Interactive Modelling with Rough Freehand 3D Sketch Strokes

Sukanya Bhattacharjee, Parag Chaudhuri

Keywords: 3D Sketches, Freeform Modelling, AR, Interactive, Supervised Learning

TL;DR: A complete end-to-end interactive framework for rapid, incremental modelling from sparse, irregular rough freehand 3D sketches.

Abstract: Freehand 3D sketches are a great medium to ideate and create visual content. However, generating 3D models from such rough sketches remains an unsolved, non-trivial task. We present a complete end-to-end interactive framework for rapid, incremental modelling from sparse, irregular 3D sketches. At the core of our solution, is 'sketchTransformer', a two-staged transformer network architecture, that fits parametric surface patches to a set of sketch strokes. We devise a novel pseudo height field representation that enables sketchTransformer to handle the noise and sparseness in the input strokes. Our method interactively evolves the surface model while maintaining smooth joins to nearby patches. We show two frontends for our framework, one on the desktop and as a mobile AR application, to illustrate how our method complements a standard 3D modelling pipelines. We can robustly handle a large variety of input 3D strokes, that competing methods cannot parse adequately.

Submission Number: 241