IconDM: Text-Guided Icon Set Expansion Using Diffusion Models

Jiawei Lin, Zhaoyun Jiang, Jiaqi Guo, Shizhao Sun, Ting Liu, Zijiang James Yang, Jian-Guang Lou, Dongmei Zhang

Abstract:

Icons are ubiquitous visual elements in graphic design. However, their creation is non-trivial and time-consuming. To this end, we draw inspiration from the booming text-to-image field and propose Text-Guided Icon Set Expansion, a task that allows users to create novel and style-preserving icons using textual descriptions and a few handmade icons as style reference. Despite its usefulness, this task poses two unique challenges. (i) Abstract Concept Visualization. Abstract concepts like technology and health are frequently encountered in icon creation, but their visualization requires a mental grounding process that connects them to physical and easy-to-draw concepts. (ii) Fine-grained Style Transfer. Unlike ordinary images, icons exhibit far richer fine-grained stylistic elements, including tones, line widths, shapes, shadow effects, etc, setting a higher demand on capturing and preserving them during generation.

To address the challenges, we propose IconDM, a method based on pre-trained text-to-image (T2I) diffusion models. It involves a one-shot domain adaptation process and an online style transfer process. The domain adaptation aims to improve the pre-trained T2I model in understanding abstract concepts by finetuning on high-quality icon-text pairs. To achieve so, we construct IconBank, a large-scale dataset of 2.3 million icon-text pairs, where the texts are generated by the state-of-the-art vision-language model from icons. In style transfer, we introduce a Style Enhancement Mod- ule into the T2I model. It explicitly extracts the fine-grained style features from the given reference icons, and is jointly optimized with the T2I model during DreamBooth tuning. To assess IconDM, we present IconBench, a structured suite with 30 icon sets and 100 concepts (including 50 abstract concepts) for generation. Quantitative results, qualitative analysis, and extensive ablation studies demonstrate the effectiveness of IconDM.

Primary Subject Area: [Content] Vision and Language

Relevance To Conference: Icons are critical elements of graphic design. We introduce a novel task where users can create icon sets using text prompts and a few reference icons, streamlining the design process and enabling both professionals and non-experts to participate. Moreover, we contribute a large-scale dataset to the community. It contains high-quality icon images with a variety of concepts and styles, which can drive future development in the field.

Submission Number: 2347