AesStyler: Aesthetic Guided Universal Style Transfer

Ran Yi, Haokun Zhu, Teng Hu, Yu-Kun Lai, Paul L Rosin

Abstract: Recent studies have shown impressive progress in universal style transfer which can integrate arbitrary styles into content images. However, existing approaches struggle with low aesthetics and disharmonious patterns in the final results. To address this problem, we propose AesStyler, a novel Aesthetic Guided Universal Style Transfer method. Specifically, our approach introduces the aesthetic assessment model, trained on a dataset with human-assessed aesthetic scores, into the universal style transfer task to accurately capture aesthetic features that universally resonate with human aesthetic preferences. Unlike previous methods which only consider aesthetics of specific style images, we propose to build a Universal Aesthetic Codebook (UAC) to harness universal aesthetic features that encapsulate the global aspects of aesthetics. Aesthetic features are fed into a novel Universal and Style-specific Aesthetic-Guided Attention (USAesA) module to guide the style transfer process. USAesA empowers our model to integrate the aesthetic attributes of both universal and style-specific aesthetic features with style features and facilitates the fusion of these aesthetically enhanced style features with content features. Extensive experiments and user studies have demonstrated that our approach generates aesthetically more harmonious and pleasing results than the state-of-the-art methods, both aesthetic-free and aesthetic-aware.

Primary Subject Area: [Experience] Art and Culture

Secondary Subject Area: [Generation] Generative Multimedia, [Content] Media Interpretation

Relevance To Conference: In this paper, we propose AesStyler, a novel Aesthetic Guided Universal Style Transfer method. Our AesStyler, by utilizing TANet as the aesthetic feature extractor, can accurately capture aesthetic features that resonate with human aesthetic preferences. Secondly, we propose to build a Universal Aesthetic Codebook (UAC) to harness universal aesthetic features that encapsulate the global aspects of aesthetics and to employ these features to guide the style transfer process. Thirdly, we propose Universal and Style-specific Aesthetic-Guided Attention (USAesA) module, which empowers our model to adaptively and progressively integrate both universal and style-specific aesthetic features with the style feature and incorporate the aes-enhanced style feature into the content feature. Extensive experiments and user studies have demonstrated the superiority of our method. Compared to state-of-the-art methods of both aesthetic-free and aesthetic-aware, AesStyler yields results of superior aesthetics and better style transfer quality.

Submission Number: 1902