Freehand Sketch Generation from Mechanical Components

Zhichao Liao, Fengyuan Piao, Di Huang, Xinghui Li, Yue Ma, Pingfa Feng, Heming Fang, Long ZENG

Abstract: Drawing freehand sketches of mechanical components on multimedia devices for AI-based engineering modeling becomes a new trend. However, its development is being impeded because existing works cannot produce suitable sketches for data-driven research. These works either generate sketches lacking a freehand style or utilize generative models not originally designed for this task resulting in poor effectiveness. To address this issue, we design a two-stage generative framework mimicking the human sketching behavior pattern, called MSFormer, which is the first time to produce humanoid freehand sketches tailored for mechanical components. The first stage employs Open CASCADE technology to obtain multi-view contour sketches from mechanical components, filtering perturbing signals for the ensuing generation process. Meanwhile, we design a view selector to simulate viewpoint selection tasks during human sketching for picking out information-rich sketches. The second stage translates contour sketches into freehand sketches by a transformer-based generator. To retain essential modeling features as much as possible and rationalize stroke distribution, we introduce a novel edge-constraint stroke initialization. Furthermore, we utilize a CLIP vision encoder and a new loss function incorporating the Hausdorff distance to enhance the generalizability and robustness of the model. Extensive experiments demonstrate that our approach achieves state-of-the-art performance for generating freehand sketches in the mechanical domain.

Primary Subject Area: [Generation] Generative Multimedia

Secondary Subject Area: [Experience] Multimedia Applications

Relevance To Conference: Nowadays with vigorous development of multimedia technology, using a digital pen to draw freehand sketches of mechanical components on multimedia devices for AI-based modeling in CAD platforms becomes a new trend. However, hindered by the scarcity of freehand sketches for mechanical components in the sketch community, the development of freehand sketch modeling has been impeded. To address this limitation, we propose a novel generative model specifically designed to produce accurate freehand sketches of mechanical components. Our work provides support for freehand sketch modeling tasks based on multimedia devices, such as component recognition, retrieval, and three-dimensional reconstruction, enhancing visualization, communication, and comprehension of intricate engineering designs. Thereby it fosters innovation and efficiency in multimedia processing applications within mechanical engineering and design domains. Additionally, our model contributes to accelerating the advancement of multimedia sketch modeling technology. With the development of these techniques, users can express ideas more intuitively and efficiently, leading to the development of more user-friendly multimedia modeling applications and interfaces, enhancing interdisciplinary collaboration in multimedia research and development.

Submission Number: 2308