PhotoStylist: Altering the Style of Photos Based on the Connotations of Texts

Siamul Karim Khan, Daniel Yue Zhang, Ziyi Kou, Yang Zhang, Dong Wang

Abstract: The need to modify a photo to reflect the connotations of a text can arise due to multifarious reasons (e.g., a musician might modify a photo in the album cover to better reflect the connotations in her song lyrics). An interesting observation is that different styles of photos convey different feelings. In this paper, we propose the PhotoStylist scheme to effectively modify the style of an input photo to represent the connotations in an input text. Existing methods that aim to transfer emotions into photos rely on an emotion class being provided as input and modify the overall color of photos based on the input emotion class, generating unrealistic colors for many objects in the image. To address these limitations, we design PhotoStylist, a novel deep-learning-based approach, to alter the individual style of each object in the photo in a way that the connotations of the input text are naturally and effectively embedded into the modified photos. Evaluation results on the Amazon Mechanical Turk (MTurk) show that our scheme can achieve output photos significantly closer to the connotations of the input text than the output photos from the state-of-the-art baselines.