Reference-Based Line Drawing Colorization Through Diffusion Model

Jiaze He, Wenqing Zhao, Ziruo Li, Jin Huang, Ping Li, Lei Zhu, Bin Sheng, Subrota Kumar Mondal

Abstract: Line drawing colorization is an indispensable stage in the image painting process, however, traditional manual coloring requires a lot of time and energy from professional artists. With the development of deep learning techniques, attempts have been made to colorize line drawings by means of user prompts, text, etc, but these methods also seem to require some manual involvement. In this paper, we propose a reference-based colorization method for cartoon line drawings, which uses a more stable diffusion model to automatically colorize line drawings to improve the quality of the generated images. In addition, to further learn the color of the reference image and improve the quality of the colorized image, we also design a two-stage training strategy. To ensure the generality of the model, in addition to the 17,769 benchmark datasets shared on the Kaggle, we used the cartoon dataset provided by the competition in the fine-tuning stage and created a small garment dataset. Finally, we illustrate the effectiveness of the model in reference-based automatic coloring through a large number of qualitative and quantitative experiments.