CraftGraffiti: Exploring Human Identity with Custom Graffiti Art via Facial-Preserving Diffusion Models

Ayan Banerjee, Fernando Vilariño, Josep Llados

Track: Paper

Keywords: Diffusion, Style Transfer, Self-Attention, Low-Rank Adaptation

TL;DR: Customizing the character in a musical scenario with graffiti style, maintaining the consistency of the face.

Abstract: Preserving facial identity under extreme stylistic transformation remains a major challenge in generative art. In graffiti, a high-contrast, abstract medium—subtle distortions to eyes, nose, or mouth can erase the subject’s recognizability, undermining both personal and cultural authenticity. We present \emph{CraftGraffiti}, an end-to-end text-guided graffiti generation framework designed with facial feature preservation as a primary objective. Given an input image and a style and pose descriptive prompt, \emph{CraftGraffiti} first applies graffiti style transfer via LoRA-fine-tuned pretrained diffusion transformer, then enforces identity fidelity through a face-consistent self-attention mechanism that augments attention layers with explicit identity embeddings. Pose customization is achieved without keypoints, using CLIP-guided prompt extension to enable dynamic re-posing while retaining facial coherence. We formally justify and empirically validate the “style-first, identity-after” paradigm, showing it reduces attribute drift compared to the reverse order. Quantitative results demonstrate competitive facial feature consistency and state-of-the-art aesthetic and human preference scores, while qualitative analyses and a live deployment at the Cruïlla Festival highlight the system’s real-world creative impact. \emph{CraftGraffiti} advances the goal of identity-respectful AI-assisted artistry, offering a principled approach for blending stylistic freedom with recognizability in creative AI applications. The code, demo, and details of the actual installation at the music Festival Cruilla 2025 in Barcelona are available at: \href{https://github.com/ayanban011/CraftGraffiti}{github.com/ayanban011/CraftGraffiti.}

Submission Number: 94