Benchmarking and Data Synthesis for Colorization of Manga Sequential Pages for Augmented Reality

Maksim Golyadkin, Sergey Saraev, Ilya Makarov

Abstract: This paper introduces an innovative approach to manga colorization within augmented reality (AR) environments, focusing on the unique challenges posed by colorizing photos of manga books. We present a novel method using diffusion models to generate a synthetic dataset that accurately replicates photographed manga pages. Additionally, we have compiled a dataset of real manga photographs, capturing diverse environmental conditions. Integrating these datasets, we established a comprehensive benchmark to evaluate colorization models in scenarios that simulate AR applications. This benchmark was validated through a human study, confirming the accuracy of our metrics across both datasets. We also showed that domain adaptation may improve model performance. Paving the way for practical applications, our framework enables the creation of an AR application designed to execute manga colorization effectively.