Towards Artist-Like Painting Agents with Multi-Granularity Semantic Alignment

Zhangli Hu, Ye Chen, Zhongyin Zhao, Jinfan Liu, Bilian Ke, Bingbing Ni

Abstract: Mainstream painting agents based on stroke-based rendering (SBR) attempt to translate visual appearance into a sequence of vectorized painting-style strokes. Lacking a direct mapping (and consequently the differentiable ability) between pixel domain and stroke parameter searching space, these methods often yield non-realistic/artist-incompatible stroke decompositions, hindering its further application in high quality art generation. To explicitly address this issue, we propose a novel SBR based image-to-painting framework which aligns with artistic oil painting behaviors/techniques. In the heart is a semantic content stratification module which decomposes images into hierarchical painting regions encapsulated with semantics, according to which a coarse-to-fine strategy is developed to first fill-in the abstract structure of the painting with coarse brushstrokes; and then depict the detailed texture portrayal with parallel-run localized multi-scale stroke search. In the meantime, we also propose a novel method that integrates SBR frameworks into a simulation-based interactive painting system for stroke quality assessment. Extensive experimental results on a wide range of images show that our method not only achieves high fidelity and artist-like painting rendering effect with a reduced number of strokes, but also exhibits greater stroke quality over prior methods.

Primary Subject Area: [Experience] Art and Culture

Secondary Subject Area: [Experience] Multimedia Applications

Relevance To Conference: Stroke based rendering is a key research area in multimedia processing, which facilitates art production, graphic design and entertainment. This technique revolves around the idea of using strokes, rather than traditional pixel-based methods, to generate images and graphics. The unique approach of SBR allows for more artistic and expressive renderings, closely mimicking the techniques used in traditional painting and drawing. This has profound implications for art production, as it opens up new avenues for digital artists to create artwork that retains the charm and style of traditional mediums. We propose an image to paint SBR algorithm based on semantic alignment that guides the process of stroke decomposition. This advancement brings the algorithm closer to human users in terms of design and understanding, laying a solid foundation for the future where users can actively participate in collaborative artistic creation with the system.

Submission Number: 404