Knowledge-Aware Artifact Image Synthesis with LLM-Enhanced Prompting and Multi-Source Supervision

Shengguang Wu, Zhenglun Chen, Qi Su

Abstract: Ancient artifacts are an important medium for cultural preservation and restoration. However, many physical copies of artifacts are either damaged or lost, leaving a blank space in archaeological and historical studies that calls for artifact image generation techniques. Despite the significant advancements in open-domain text-to-image synthesis, existing approaches fail to capture the important domain knowledge presented in the textual description, resulting in errors in recreated images such as incorrect shapes and patterns. In this paper, we propose a novel knowledge-aware artifact image synthesis approach that brings lost historical objects accurately into their visual forms. We use a pretrained diffusion model as backbone and introduce three key techniques to enhance the text-to-image generation framework: 1) we construct prompt with explicit archeological knowledge elicited from large language models (LLMs); 2) we incorporate additional textual guidance to correlated historical expertise in a contrastive manner; 3) we introduce further visual-semantic constraints on edge and perceptual features that enable our model to learn more intricate visual details of the artifacts. Compared to existing approaches, our proposed model produces higher-quality artifact images that align better with the implicit details and historical knowledge contained within written literature.

Primary Subject Area: [Experience] Art and Culture

Secondary Subject Area: [Experience] Multimedia Applications, [Generation] Generative Multimedia

Relevance To Conference: This work addresses the practical challenge of lost historical artifacts from the perspective of multimodal generative AI. We propose novel diffusion-based algorithmic designs tailored for accurate artifact image synthesis that incorporate both expert archaeological knowledge and fine-grained vision-language constraints. Our method accurately revisualizes historical artifacts, aiding research in historical Art and Culture as well as Generative Multimedia.

Submission Number: 4527